Chapter 2 Introduction to Neural network

Assume that we have a sequence of input vectors and a corresponding
sequence of target (desired) scalars
(¯x 1 , t 1 ), (¯x 2 , t 2 ), , · · · , (¯x N , t N )
We wish to find the weights of a neuron with a non-linear function
f(·) so that we can minimize the squared difference between the
output y n and the target t n , i.e.
min E n = min 1 2 (t n − y n ) 2 = min 1 2 e2 n
, n = 1, 2, · · · , N
We will use the steepest descent approach
¯w (n+1) = ¯w (n) − α∇ w E
We need to find ∇ w E!
1
∇ w E n = ∇ w
2 (t n − f( ¯w T ¯x } {{ n ) }
u
} {{ }
f(u)
} {{ }
h(f)
) 2
} {{ }
g(h)
The chain-rule gives ∇ w g(h(f(u))) = ∇ h g∇ f h∇ u f∇ w u
∇ h g = e n
∇ f h = t n − f(u) = −1
∇ u f = depends on the nonlinearity f(·) we choose
∇ w u = ¯x n
⇒ ¯w (n+1) = ¯w (n) + αe n ∇ u f¯x n
since f(·) is a function f : R → R, i.e. one-dimensional. We can
write ∇ u f = df
du
⇒ The neuron learning rule for general function f(·) is
¯w (n+1) = ¯w (n) + αe n
df
du¯x n
Where u = ¯w T ¯x and α is the stepsize.
OBS! If f(u) = u, that is a linear function with slope 1, the above
algorithm will become the LMS alg. for a linear neuron.
( df
du = 1 )
42