Chapter 2 Introduction to Neural network

Main problem is the steepest descent algorithm. We know that
steepest descent is perpendicular to the contours of the error function.
closest path
We also know that the more elliptic the contour is the slower the
convergence will be. The ratio of the ellipses diameters are controlled
by the eigenvalues of the correlation matrix R xx (x-input).
If we decorrelate the input before the network we will get better
performance.
λ
2 λ
1
λ− eigenvalues of R xx
Decorrelation can be done by an unsupervised algorithm (The Generalized
Hebbian algorithm) by finding the Principal Components
(PCA). We will se this later in this course.
6.8 Momentum update
In order to drop faster in the error surface we use the average of
previous gradients when we update the weights
△w (k)
ij
= − ∂E + β△w (k−1)
ij
∂w ij } {{ }
momentum term, β