Evolution and Optimum Seeking

70 Hill climbing Strategies
with the correction factor
(k) = rF (x (k) ) T rF (x (k) )
rF (x (k;1) ) T rF (x (k;1) )
For a quadratic objective function with a positive de nite Hessian matrix, conjugate
directions are generated in this way and the minimum is found with n line searches. Since
at any time only the last direction needs to be stored, the storage requirement increases
linearly with the number of variables. This often signi es a great advantage over other
strategies. In the general, non-linear, non-quadratic case more than n iterations must be
carried out, for which the method of Fletcher and Reeves must be modi ed. Continued
application of the recursion formula (Equation (3.25)) can lead to linear dependence of
the search directions. For this reason it seems necessary to forget from time to time the
accumulated information and to start afresh with the simple gradient direction (Crowder
and Wolfe, 1972). Various suggestions have been made for the frequency of this restart
rule (Fletcher, 1972a). Absolute reliability of convergence in the general case is still not
guaranteed by this approach. If the Hessian matrix of second partial derivatives has
points of singularity, then the conjugate gradient strategy can fail. The exactness of
the line searches also has an important e ect on the convergence rate (Kawamura and
Volz, 1973). Polak (1971) de nes conditions under which the method of Fletcher and
Reeves achieves greater than linear convergence. Fletcher (1972c) himself has written a
FORTRAN program.
Other conjugate gradient methods have been proposed by Powell (1962), Polak and
Ribiere (1969) (see also Klessig and Polak, 1972), Hestenes (1969), and Zoutendijk (1970).
Schley (1968) has published a complete FORTRAN program. Conjugate directions are
also produced by theprojected gradient methods (Myers, 1968 Pearson, 1969 Sorenson,
1969 Cornick and Michel, 1972) and the memory gradient methods (Miele and Cantrell,
1969, 1970 see also Cantrell, 1969 Cragg and Levy, 1969 Miele, 1969 Miele, Huang,
and Heidemann, 1969 Miele, Levy, and Cragg, 1971 Miele, Tietze, and Levy, 1972
Miele et al., 1974). Relevant theoretical investigations have been made by, among others,
Greenstadt (1967a), Daniel (1967a, 1970, 1973), Huang (1970), Beale (1972), and Cohen
(1972).
Conjugate gradient methods are encountered especially frequently in the elds of
functional optimization and optimal control problems (Daniel, 1967b, 1971 Pagurek and
Woodside, 1968 Nenonen and Pagurek, 1969 Roberts and Davis, 1969 Polyak, 1969
Lasdon, 1970 Kelley and Speyer, 1970 Kelley and Myers, 1971 Speyer et al., 1971
Kammerer and Nashed, 1972 Szego and Treccani, 1972 Polak, 1972 McCormick and
Ritter, 1974). Variable metric strategies are also sometimes classi ed as conjugate gradient
procedures, but more usually as quasi-Newton methods. For quadratic objective
functions they generate the same sequence of points as the Fletcher-Reeves strategy and
its modi cations (Myers, 1968 Huang, 1970). In the non-quadratic case, however, the
search directions are di erent. With the variable metric, but not with conjugate directions,
Newton directions are approximated.
For many practical problems it is very di cult if not impossible to specify the partial
derivatives as functions. The sensitivity of most conjugate gradient methods to imprecise