AI - a Guide to Intelligent Systems.pdf - Member of EEPIS

186
ARTIFICIAL NEURAL NETWORKS
Figure 6.14
Learning with momentum
accelerate descent in the steady downhill direction, and to slow down the
process when the learning surface exhibits peaks and valleys.
Figure 6.14 represents learning with momentum for operation Exclusive-OR.
A comparison with a pure back-propagation algorithm shows that we reduced
the number of epochs from 224 to 126.
In the delta and generalised delta rules, we use a constant and rather
small value for the learning rate parameter, a. Can we increase this value
to speed up training?
One of the most effective means to accelerate the convergence of backpropagation
learning is to adjust the learning rate parameter during training.
The small learning rate parameter, , causes small changes to the weights in the
network from one iteration to the next, and thus leads to the smooth learning
curve. On the other hand, if the learning rate parameter, , is made larger to
speed up the training process, the resulting larger changes in the weights may
cause instability and, as a result, the network may become oscillatory.
To accelerate the convergence and yet avoid the danger of instability, we can
apply two heuristics (Jacobs, 1988):
. Heuristic 1. If the change of the sum of squared errors has the same algebraic
sign for several consequent epochs, then the learning rate parameter, ,
should be increased.
. Heuristic 2. If the algebraic sign of the change of the sum of squared errors
alternates for several consequent epochs, then the learning rate parameter, ,
should be decreased.