The Development of Neural Network Based System Identification ...

4.2 THE ARTIFICIAL NEURAL NETWORKS 89
connections from context units to hidden units. Figure 4.6(c) shows the architecture of
the modified Elman network with reduced weight connections. Note that the connections
from context units to hidden units are made with one to one connections instead of
multiple connections from a single context unit to multiple hidden units as in Figure
4.6(b). For a single hidden layer case, the outputs formulation from hidden, context
unit and output layer of a modified Elman network are given as follows:
v h (t) = g h
⎛
⎝
m∑
W 1 hj X j (t) + B1 h +
j=1
⎞
h∑
W 3 k x k (t) ⎠ for h = 1, 2, 3 · · · H (4.8)
k=1
x k (t) = v h (t − 1) + αx k (t − 1) for k = 1, 2, 3 · · · H (4.9)
( H
)
∑
ŷ i (t) = g i W 2 ih v h (t) + B2 i for i = 1, 2, 3 · · · n (4.10)
h=1
Similar to MLP and HMLP network architecture, variables m, n and H denote the
number of inputs, outputs and hidden nodes of the Elman network respectively, B1 h
and B2 i is the bias elements of input and output layer and X j (t) denotes the inputs data
fed into the Elman network. The function g h and g i are the activation functions used in
the hidden and output layer similar to MLP architecture. The weight connections that
connects the input layer to the hidden layer is given by matrix W 1 hj . Matrix W 2 ih
indicates weight matrix that connect the hidden layer to the output layer. The weight
connections from context units to hidden neurons are represented by vector W 3 k . The
self-connections α in the context units are set to be the same for all context units and
typically selected between 0 ≤ α ≤ 1. The higher the value of α indicates that network
has the higher capability to trace the gradient further into the past [Pham and Liu,
1993].