The Development of Neural Network Based System Identification ...

4.3 SYSTEM IDENTIFICATION WITH NEURAL NETWORK 103
In order to minimise the cost function in Equation (4.28), the Levenberg-Marquardt
(LM) iterative search algorithm was used for the neural network training process. The
LM algorithm is considered the best choice of training method in many NN applications
particularly for training small to medium size networks [Naghdinezhad et al., 2006,
Wilamowski and Hao, 2010, Wilamowski, 2009, Garratt and Anavatti, 2012]. The
problems of criterion minimisation in NN are often ill-conditioned as reviewed in
Saarinen et al. [1993] and Ngia and Sjoberg [2000]. As a remedy to the problem, the
LM algorithm adds positive diagonal matrix λI to improve the numerical conditioning
to the Hessian matrix [Norgaard, 2000, Ngia and Sjoberg, 2000]. Typically, the LM
optimisation process is carried out iteratively over a given data set to achieve the
minimum error criterion. The LM optimisation algorithm uses the Gauss-Newton’s
Gradient vector G(θ) and Hessian matrix R(θ), which were derived specifically for the
neural network model in Yu and Wilamowski [2011], Norgaard [2000]. These important
matrices are represented in the following equations:
G (θ) = V ′ N (θ, Z N ) = 1 N
R (θ) = V ′′
N (θ, Z N ) = 1 N
N∑
ψ (t|θ) [y (t) − ŷ (t|θ)] (4.31)
t=1
N∑
ψ (t|θ) [ψ (t|θ)] T (4.32)
where ψ(t|θ) is the matrix that represent the first order partial derivative of the one-step
ahead prediction with respect to the parameter vector θ and it is also known as the
Jacobian matrix. The complete formulations of Jacobian matrix for MLP, HMLP and
Elman networks are given in Section 4.3.4.1. The Hessian matrix R(θ) has a dimension
of d × d and Gradient vector G(θ) with dimension of d × 1, where d is the total number
t=1
of elements (weights + biases) in the parameter vector θ.
For the LM iterative algorithm, the step size µ (i) in Equation (4.30) is set to µ (i) = 1
and the H (i) matrix is replaced with the following equation:
H (i) = R (i) (θ) + λ (i) I (4.33)
where R (i) (θ) is defined as V ′′
N (θ, Z N) in Equation (4.32). Finally, we could find the