Artificial Intelligence and Soft Computing: Behavioral ... - Arteimi.info

Trapping at local minima: Back-propagation adjusts the weights to reach
the minima (vide fig. 14.8) of the error function (of weights). However, the
network can be trapped in local minima. This problem, however, can be solved
by adding momentum to the training rule or by statistical training methods
applied over the back-propagation algorithm [26].
Weight 2
Error
function
of
weights
local
minima1
Local
minima2
Weight 1
Fig. 14.8:Valleys in error function cause back-propagation algorithm trapped at
local minima.
Adding momentum to the training rule: To eliminate the problem of
trapping at local minima, recently a momentum term was added to the right
hand side of the adaptation rule [17]. Formally,
Wp,q,k(n+1) = Wp,q,k(n) + η δq, k Outp, j + α ∆ wp, q, k (n-1).
The last term in the right hand side corresponds to momentum. The addition of
the momentum term forces the system to continue moving in the same
direction on the error surface, without trapping at local minima. A question
may naturally arise: why call this momentum term? The answer came from the
analogy of a rolling ball with high momentum passing over a narrow hole.