The Development of Neural Network Based System Identification ...

106 CHAPTER 4 NEURAL NETWORK BASED SYSTEM IDENTIFICATION
of the Jacobian matrix ψ(t|θ) would become d × (nN) if multiple input-output variables
were considered in the estimation problem. The Jacobian matrix ψ(t|θ) also can be
effectively calculated using an alternative approach such as finite differences method
[Norgaard, 2000, Yu and Wilamowski, 2011, Wilamowski et al., 2008].
Consider a two layer MLP network with hyperbolic tangent hidden units and linear
output units:
ŷ i (t | θ) mlp =
=
⎛ ⎛
⎞ ⎞
H∑
m∑
⎝W 2 ih tanh ⎝ W 1 hj ϕ j (t) + B1 h
⎠ + B2 i
⎠
h=1
j=1
H∑
W 2 ih v h (t) + B2 i
h=1
with h = 1, 2, 3 · · · H and i = 1, 2, 3 · · · n (4.36)
The MLP Jacobian matrix ψ(t|θ) is calculated by partial differentiating Equation (4.36)
with respect to parameter vector θ arriving at the following results:
⎧
v h (t)
if θ = W 2 ih
ψ (t|θ) mlp
=
∂ŷ (t|θ)
∂θ
⎪⎨
=
⎪⎩
1 if θ = B2 i
(
W 2 ih 1 − v
2
h
(t) ) ϕ j (t)
(
W 2 ih 1 − v
2
h
(t) )
if θ = W 1 hj
if θ = B1 h
0 otherwise
(4.37)
The output prediction from the HMLP can also be expressed with the hyperbolic
tangent hidden units and linear output unit as follows:
ŷ i (t |θ ) hmlp
=
=
⎛ ⎛
⎞ ⎞
H∑
m∑
⎝W 2 ih tanh ⎝ W 1 hj ϕ j (t) + B1 h
⎠ + B2 i
⎠ +
h=1
j=1
H∑
(W 2 ih v h (t) + B2 i ) +
h=1
m∑
W 3 ij ϕ j (t)
j=1
m∑
W 3 ij ϕ j (t)
with h = 1, 2, 3 · · · H and i = 1, 2, 3 · · · n (4.38)
j=1
where W 3 ij is the weights matrix of linear connection between input layer and output