Artificial Intelligence and Soft Computing: Behavioral ... - Arteimi.info

estimated through a forward pass [21] on the network. After the forward pass
is over, the error vector at the output layer is estimated by taking the
component-wise difference of the target pattern and the generated output
vector. A function of errors of the output layered nodes is then propagated
back through the network to each layer for adjustment of weights in that layer.
The weight adaptation policy in back-propagation algorithm is derived
following the principle of steepest descent approach [16] of finding minima
a multi-valued function [5]. A derivation of the algorithm is given in
B.
The most significant issue of a back-propagation algorithm is the
propagation of error through non-linear inhibiting function in backward
direction. Before this was invented, training with a multi-layered feed-forward
neural network was just beyond imagination. In this section, the process of
propagation of error from one layer to its previous layer will be discussed
shortly. Further, how these propagated errors are used for weight adaptation
will also be presented schematically.
Typical neurons employed in back-propagation learning contain two
modules (vide fig. 14.6(a)). The circle containing ∑ wi xi denotes a weighted
sum of the inputs xi for i= 1 to n. The rectangular box in fig. 14.6(a) represents
the sigmoid type non-linearity. It may be added here that the sigmoid has been
chosen here because of the continuity of the function over a wide range. The
continuity of the nonlinear function is required in back-propagation, as we
have to differentiate the function to realize the steepest descent criteria of
learning. Fig. 14.6(b) is a symbolic representation of the neurons used in fig.
14.6(c ).
In fig. 14.6(c ), two layers of neurons have been shown. The left side
layer is the penultimate (k –1)-th layer, whereas the single neuron in the next
k-th layer represents one of the output layered neurons. We denote the top two
neurons at the (k-1)-th and k-th layer by neuron p and q respectively. The
connecting weight between them is denoted by wp,q,k. For computing
Wp,q,k(n+1), from its value at iteration n, we use the formula presented in
expression (14.2-14.4).
We already mentioned that a function of error is propagated from the
nodes in the output layer to other layers for the adjustment of weight. This
functional form of the back-propagated error is presented in expression (14.4)
and illustrated in fig. 14.7. It is seen from expression (14.4) that the
contribution of the errors of each node at the output layer is taken into account
in an exhaustively connected neural net.
For training a network by this algorithm, one has to execute the
following 4 steps in order for all patterns one by one.