The Development of Neural Network Based System Identification ...

5.2 OFF-LINE BASED SYSTEM IDENTIFICATION FOR MLP NETWORK 123
Weights
60
40
20
0
MSE
10 −2
10 −3
Validation
Training
10 −1 Iterations
−20
−40
0 50 100 150 200 250 300 350 400 450 500
Iterations
(a)
10 −4
0 50 100 150 200 250 300 350 400 450 500
(b)
Weights
10
5
0
−5
MSE
10 −2
10 −3
Validation
Training
10 −1 Iterations
−10
0 50 100 150 200
Iterations
(c)
10 −4
0 50 100 150 200
(d)
Weights
1.5
1
0.5
0
−0.5
−1
MSE
10 −1
10 −2
10 −3
Validation
Training
10 0 Iterations
−1.5
0 20 40 60 80 100 120 140 160 180
Iterations
(e)
10 −4
0 20 40 60 80 100 120 140 160 180
(f)
Figure 5.3 The prediction performance of MLP network with varying regularisation parameter α.
(a), (c) and (e) show the weights adaptation during the training process where each line represents single
weight in the network. The error evaluation on the training and validation datasets is shown in (b), (d)
and (f): (a) Network weights adaptation for α = 0 (no regularisation term in the error cost function);
(b) Training progress for α = 0; (c) Network weights adaptation for α = 0.0001; (d) Training progress
for α = 0.0001; (e) Network weights adaptation for α = 0.8; and (f) Training progress for α = 0.8.
lead to an over-fitting problem as indicated in the MSE plot. The optimum weights
are among the smallest at this point compared with those beyond iteration i = 112 as
shown in Figure 5.3(a). This figure also demonstrates that without regularisation term,
some network weights would grow to a large magnitude causing the network to over-fit.
Figure 5.3(d) shows that the training with α = 0.0001 completes in 246 iterations,
and the optimum weights are found at iteration i = 185. The weights adaptation plot