The Development of Neural Network Based System Identification ...

4.2 THE ARTIFICIAL NEURAL NETWORKS 83
ŷ1
yˆn
Output Layer
B1
W2n,h
1
Bias Input
Bn
1
W21,1 W2n,1
W2n,2 W21,h
W21,2
2
h
Hidden Layer
b1
b2
1
Bias Input
bh
W11,1
W12,1 W12,m
W1h,1
W11,m
W1h,m
Input Layer
X1
Xm
Figure 4.4 A fully connected, feed-forward multi-layered perceptron (MLP) with m-inputs, h-hidden
neurons and n-outputs.
more hidden layers in regression problem [Tu, 1996]. Moreover, Funahashi [1989] and
Cybenko [1989] have proven that any non-linear relationship function mapping can be
sufficiently approximate with a single hidden layer with sigmoid activation function.
The MLP architecture with one hidden layer is shown in Figure 4.4. For a single hidden
layer case, the outputs formulation from hidden and output layer of a MLP network is
given as follows:
v h (t) = g h
⎛
⎝
⎞
m∑
W 1 hj X j (t) + b h
⎠ ; for h = 1, 2, 3 · · · H (4.1)
j=1
( H
)
∑
ŷ i (t) = g i W 2 ih V h (t) + B i ; for i = 1, 2, 3 · · · n (4.2)
h=1
where W 1 hj is the weights matrix between the input layer and the hidden layer and
W 2 ih is the weights matrix between the hidden layer and the output layer. The functions
g h (∗) and g i (∗) are non-linear activation function for neurons in each hidden and output
layers. The symbol H denotes the number of neurons in the hidden layer while b h and
B i are the bias elements for the input layer and output layer. The number of inputs
and outputs of neural network are presented by m and n respectively. In Equation (4.1)
and (4.2), the weight connections and biases in the network structure are included in