The Development of Neural Network Based System Identification ...

110 CHAPTER 4 NEURAL NETWORK BASED SYSTEM IDENTIFICATION
V N (θ, Z N ) in Equation (4.28) such as:
W N (θ, Z N ) = 1
2N
N∑
t=1
[y (t) − ŷ (t |θ )] 2 + 1
2N θT Dθ (4.44)
Matrix D is a diagonal matrix which is often selected as D = αI (α > 0) or D = 0,
where α denotes weight decay or regularisation parameter that controls the amount of
regularisation introduced to the criterion V N (θ, Z N ). The larger the α value, the more
important the regularisation becomes. The regularisation method prevents the weight
from getting larger by minimising the sum square error and the regularisation term.
Apart from augmenting the criterion with regularisation term, the LM algorithm
also needs several more modifications to match the regularised criterion by adding
additional regularisation terms to the Gradient and Hessian matrix:
G (θ) = W ′ N (θ, Z N ) = 1 N
R (θ) = W ′′
N (θ, Z N ) = 1 N
N∑
t=1
ψ (t|θ) [y (t) − ŷ (t|θ)] + 1 Dθ (4.45)
N
N∑
ψ (t|θ) [ψ (t|θ)] T + 1 N D (4.46)
t=1
The ratio r (i) for updating the parameter λ (i) apparently needs to be changed to:
r (i) = 2 [ (
V N θ (i) ) (
, Z N − VN θ (i) + f (i) )]
, Z N
(
f (i)) ( T (
G θ (i)) +
[λ (i) I + 1 ] )
N D f (i)
} {{ }
reduction approximation
(4.47)