Chapter 2 Introduction to Neural network

The backpropagation algorithm can cause the weight update to follow
a fractal pattern (fractal means it follow some rules which are
applied statistical), (see figure 8.11)
⇒ The stepsize and momentum term should decrease during training
to avoid this.
6.9 Adaptive step algorithms
These algorithms modifies the stepsize during training in order to
speed convergence. The idea is to increase the stepsize,α , if the
sign of the gradient is the same for two following updates and to
decrease if the sign change.
Example: Error function
start
increase
decrease
increase
6.10 Silva and Almeidal’s algorithms
α (k+1)
i =
{
α (k)
α (k)
i u if ∇ i E (k) ∇ i E (k−1) > 0
i d if ∇ i E (k) ∇ i E (k−1) < 0
u, d parameters (e.g. u = 1.1, d = 0.9) and index i stands for weight
no. i
□
Other similar methods are Delta-bar-delta and and Rprop.
6.11 Second order algorithms
These methods use the Newton’s algorithm instead of the steepest
descent.
[ ] ∂
w (k+1) = w (k) 2 −1
E
− ∇
∂w 2 w E
54