The Development of Neural Network Based System Identification ...

80 CHAPTER 4 NEURAL NETWORK BASED SYSTEM IDENTIFICATION
The computational model of a single neuron that can perform intelligent functions
or tasks is shown in 4.2(a). As an example, the process to produce a pulse-width output
signal inside a single neuron is shown in detail in Figure 4.2(b). The processing unit or
neuron accepts external inputs through the weighted connections, W hj and all of these
signals are summed up before being fed to the activation function. External inputs such
as data measurement or outputs from the other neuron’s calculation can be supplied to
the neuron for processing. If the activation function in the neuron is a linear function,
this NN model is also known as the adaptive linear neuron model (ADALINE) which
was first developed by Widrow and Hoff [1960]. The learning of the ADALINE network
was done by minimising the square error between the target value and the output of
the model. This learning process is also known as gradient descent in neural learning or
least square error minimisation in statistical methods [Samarasinghe, 2007].
The activation function, f h in the neurons is used to introduce non-linearity into
the network. The activation function is typically selected as non-linear and continuous
function that remains bounded within some upper and lower bounds [Norgaard, 2000,
Samal, 2009]. Since the output of non-linear function varies non-linearly with the input,
the NN model can perform non-linear mapping between inputs and outputs. Hornik
et al. [1989], Tu [1996] and Tu [1996] further proved that a non-linear activation function
such as sigmoid function introduced in the neurons would make the NN capable of
approximating any non-linear function of interest to any desired degree of accuracy
with sufficiently available neurons. Hornik et al. [1989] further implies that any failure
of a function mapping by a multilayer network must arise from inadequate choice of
weight parameters or an insufficient number of hidden nodes.
The versatility of a NN as a universal approximator is not only limited to sigmoid
function. Stinchcombe and White [1989] proved in their work that NN with general class
of non-linear function can also achieve universal approximation. Furthermore, the nonlinear
function mapping can also be approximated by NN with a bell shaped activations
function in the hidden neuron unit [Baldi, 1990]. Different activation functions are
used in neural networks training and some of the activation examples are given in
Figure 4.3. The step and sign functions are activation function typically used for binary