The Development of Neural Network Based System Identification ...

104 CHAPTER 4 NEURAL NETWORK BASED SYSTEM IDENTIFICATION
minimum of the error criterion by iteratively solving the following update equation:
[
−1
θ (i+1) = θ (i) − R (i) (θ) + λ I] (i) G (i) (θ) (4.34)
where I is the identity matrix with a size equal to Hessian R (θ) matrix and the λ (i)
constant is a damping factor used for deciding the step size. The value of λ (i) is selected
to be λ (i) ≥ 0. The usage of the constant λ (i) can also be viewed as a blending factor
between Gauss-Newton (GN) and Steepest Descent (SD) update, which provides LM
algorithm with fast convergence speed of the GN algorithm and the stability of SD
method [Yu and Wilamowski, 2011]. The largest update can be possibly achieved by
choosing λ (i) = 0 which gives a full GN step. Shorter step length as in SD algorithm
is achieved by taking λ (i) → ∞ which will cause the diagonal elements of R (i) (θ) to
dominate [Ngia and Sjoberg, 2000].
In order to determine λ, the indirect method used in Norgaard [2000] and Fletcher
[1981] is adopted by calculating the following ratio to determine the accuracy of
approximation:
r (i) = 2 [ (
V N θ (i) ) (
, Z N − VN θ (i) + f (i) )]
, Z N
(f (i)) T (
G θ (i)) (
+ f (i)) T
f (i) λ (i)
} {{ }
reduction approximation
(4.35)
The main purpose of introducing the ratio calculation is to measure how well the
reduction of the criterion V N (θ, Z N ) matches the reduction predicted by approximation
terms in denominator of ratio calculation in Equation (4.35). The damping factor λ is
adjusted accordingly to the ratio r (i) by some factor [Norgaard, 2000]. The procedure
for the LM algorithm using indirect method to determine λ is given in Figure 4.11. The
reduction approximation (denominator term in Equation (4.35)) is most likely a close
approximation to error criterion V N (θ, Z N ), if the ratio r (i) value is close to one and
parameter λ should be reduced by some factor. However, if the ratio r (i) is small or
a negative value, parameter λ should be increased. Additional stopping criteria are
normally introduced to this algorithm to prevent minimisation problems or to force
early stopping such as stopping criteria based on maximum number of iterations, sum
of square error that drops below a certain threshold, upper bound for gradient and