The Development of Neural Network Based System Identification ...

108 CHAPTER 4 NEURAL NETWORK BASED SYSTEM IDENTIFICATION
Pham and Liu [1996] has suggested that the internal feedback x k (t) has dependency on
the previous weights W 3 k (t − 1). This needs to be taken into account by updating the
term ∂v h (t)/∂W 3 k (t − 1) using the following formulation:
∂v h (t)
∂W 3 k (t − 1) = ( 1 − vh 2 (t)) v h (t − 1) + α ∂v h(t − 1)
∂W 3 k (t − 1)
(4.42)
where α denotes the value of the self-connection in the context units. If the weight W 3 k
changes are assumed to be small in each iteration, then the term ∂v h (t)/∂W 3 k (t − 1)
can be approximately written in recursive form as:
∂v h (t)
∂W 3 k (t − 1) = ( 1 − vh 2 (t)) v h (t − 1) + α ∂v h(t − 1)
∂W 3 k (t − 2)
(4.43)
4.3.4.2 Training by Weight Regularisation
Implementation of NN model to predict or extract usable pattern from data requires
special attention to several important aspects in model development such as examination
of the generalisation ability, minimising model complexity, testing robustness of the
model and selecting relevant inputs [Samarasinghe, 2007, Norgaard, 2000, May et al.,
2011]. The generalisation performance of the trained NN model is analysed through the
use of a validation data set which differs from the data set used for the training process.
Generalisation indicates how well a trained model performs on a new data set. It is
particularly important if a reliable prediction quality for new data is desired.
One of the main problems that may occur during neural network training is overfitting
[Sjoberg and Ljung, 1995]. This particular problem can be observed in network
training where the error of the training data set could be reduced to a very small
value, but when a new data set (validation data) is presented to the same model,
the prediction error increase. The over-fitting problem usually happens due to the
contribution of variance error which indicates an excessive number of neurons/weights
in the network. On the other hand, if the model contains insufficient neurons or weights
(under parametrised network), the bias error would dominate in such a situation. The
situation in which model prediction under-fit (bias) and over-fit the validation data