The Development of Neural Network Based System Identification ...

4.3 SYSTEM IDENTIFICATION WITH NEURAL NETWORK 109
Prediction
Data
Prediction
Data
f(x)
f(x)
x
(a)
x
(b)
Prediction
Data
f(x)
x
(c)
(d)
Figure 4.12 The bias-variance dilemma in prediction function: (a) Prediction from a model that
generalise well on validation data; (b) Prediction from a model that over-fit the data due to excessive
amount of free parameters (weights); (c) Prediction from a model that under-fit due to limited amount of
free parameter (weights); and (d) A visualisation of the bias-variance error dilemma in model prediction.
Figure adapted from Wang et al. [2008].
with noise is known as bias-variance dilemma which is shown in Figure 4.12. The effect
of over-fitting, under-fitting and good generalisation performance of a prediction model
is illustrated in Figure 4.12(a)–4.12(c). The overall description of effect of weights
dimension on generalisation error is given in Figure 4.12(d). Thus, in order to obtain
a reliable NN model with reduced generalisation error, a trade-off between these two
extreme situations needs to be addressed.
To satisfy the bias-variance dilemma, the regularisation method has been proposed
in this work to improve the generalisation ability of the neural network model. Regularisation
method is an approach proposed to control the growth of weights during training
to avoid over-fitting problems without relying on early stopping criterion [Sjoberg and
Ljung, 1995, Samarasinghe, 2007, Norgaard, 2000]. The approach attempts to limit the
flexibility of the network by introducing a regularisation term to augment the criterion