download searchable PDF of Circuit Design book - IEEE Global ...

136 Gradient Optimization
The Fletcher-Reeves formula is quite simple. As is common practice, the first
search direction is the negative gradient. Then, each new search direction is a
linear combination of the current gradient and the last search direction; the
amount of the latter j~ scaled in proportion to the squared ratio of magnitudes
of the current and last gradients. Derivation of the {3i scale factor is given in
Appendix C. Only three vectors must be stored at a time: the x variables, the s
search direction, and the g gradient components.
Example 5.4. Example 5.1 was a line search in the negative-gradient direction.
ft will now be shown that Example 5.3 illustrated a second search
direction to the global minimum that happens to agree with the Fletcher­
Reeves formula. Program A5-2 shows that at the first turning point, x =
(5.1940, S.6543l, the gradient is g=(- 11.4990, 41.071Sl. The last gradient at
point p=(\O, IOl was (-100, -28)T. Equation (5.55) shows that ,8,=0.1687;
thus (5.54) yields a new search direction: s'= (-5.3675, - 45.7929l. A second
linear search in this direction would find that 0,=0.0361, as in (5.39). Thus
0,5':=( -0.1940, - 1.6543)T, as already found by other means in Example 5.3.
Convergence will not be achieved in just N linear searches on nonquadratic.
surface~. The Fletcher-Reeves policy is to periodically restart the search
direction sequence with the current negative gradient direction. An effective
choice is to generate N directions by (5.54) and then start over again with the
negative gradient. This has been justified experimentally by many researchers.
5.1.5. Summary of Conjugate Gradient Search. Linear searches have been
described, and three strategies for selecting their sequence of directions have
been discussed. The relaxation (one-at-a-time) method was shown not to be
generally effective; however, it is significant because it works well on ellipsoids
without cross-variable terms such as x\x 2 • etc. The steepest-descent strategy is
effective far from a minimum but tends to zigzag badly in curved valleys. The
conjugate gradient method lends to follow curved valleys better, since it uses
prior gradient information to moderate zigzagging.
Several additional properties of quadratic functions were discussed to
clarify choices and introduce some concepts that are likely to be encountered
in the field of nonlinear programming. The concept of diagonalizing a
quadratic form, Le., making a linear change of variables to obtain alignment
with the ellipsoidal axes, amounts to justification for the application of
A-conjugacy in search direction-selection. ft also shows the clear possibility
for quadratic termination: the sequence of N linear searches to exact minima
in N-variable space so that the global quadratic minimum is found. The
constant nature of the mapping of variable to gradient space for quadratic
functions was mentioned because of its close relationship to Newton's method
and the variable metric search scheme. Davidon's use of the inverse Hessian
matrix as a metric for gradients leads to a simple expression for quadratic
function elevation above the global minimum. ft is also the basis for naming