The Development of Neural Network Based System Identification ...

34 CHAPTER 2 LITERATURE REVIEW
Neural Network Model
Target Output
Training Data
Prediction
Compare
Error
Signal
NN Weights
Modification
Learning/Training
Algorithm
Figure 2.6
The representation of neural network training process.
The choice of training algorithm is an important factor to consider before the NN
model is trained for prediction tasks. Empirical findings indicated that NN models
trained with the back-propagation or steepest descent (SD) method were unable to
generalise well with new observations and even failed to match the prediction quality of
the first principle model. It is well known that steepest descent algorithm has several
disadvantages compared with second order algorithms such as slow convergence rate;
and the minimisation search can easily be trapped in local minima [Billings et al., 1991,
Mashor, 2003, Wilamowski, 2011a].
Among all mentioned training methods, the LM optimisation algorithm is the most
widely used training method for off-line NN training. The algorithm is capable of
providing fast solution convergence, robust and simple to implement. The convergence
of the LM algorithm is shown to be faster than other training methods [Samarasinghe,
2007, Garratt and Anavatti, 2012, Demuth and Beale, 2000]. However, the computation
time of the LM algorithm increases as the size of the network grows. As the network
becomes larger, the Jacobian and Hessian matrices that need to be calculated and stored
in memory increase. Subsequently, this leads to increased computational time of the
LM algorithm due to memory limitation [Wilamowski and Hao, 2010]. This prevents
the LM algorithm from being implemented for large networks or modelling problems
that involve an enormous size of training data.