Evolution and Optimum Seeking

66 Hill climbing Strategies
A distinction is sometimes drawn between short step methods, which evaluate the gradients
again after a small step in the direction rF (x (k) ) (for maximization) or ;rF (x (k) )
(for minimization), and their equivalent long step methods. Since for nite step lengths
s (k) it is not certain whether the new variable vector is really better than the old, after the
step the value of the objective function must be tested again. Working with small steps
increases the number of objective function calls and gradientevaluations. Besides F (x) n
partial derivatives must be evaluated. Even if the slopes can be obtained analytically and
can be speci ed as functions, there is no reason to suppose that the number of computational
operations per function call is much less than for the objective function itself.
Except in special cases, the total cost increases roughly as the weighting factor (n +1)
and the number of objective function calls. This also holds if the partial derivatives are
approximated by di erential quotients obtained by means of trial steps
Fxi(x) = @F(x)
=
@xi
F (x + ei) ; F (x) 2
+ O( ) for all i =1(1)n
Additional di culties arise here since for values of that are too small the subtraction
is subject to rounding error, while for trial steps that are too large the neglect of terms
O( 2 ) leads to incorrect values. The choice of suitable deviations requires special care
in all cases (Hildebrand, 1956 Curtis and Reid, 1974).
Cauchy (1847), Kantorovich (1940, 1945), Levenberg (1944), and Curry (1944) are the
originators of the gradient strategy, which started life as a method of solving equations
and systems of equations. It is rst referred to as an aid to solving variational problems
by Hadamard (1908) and Courant (1943). Whereas Cauchy works with xed step lengths
s (k) , Curry tries to determine the distance covered in the (not normalized) direction
v (k) = ;rF (x (k) ) so as to reach a relative minimum (see also Brown, 1959). In principle,
any one of the one dimensional search methods of Section 3.1 can be called upon to nd
the optimal value for s (k) :
F (x (k) + s (k) v (k) ) = min
s fF (x (k) + sv (k) )g
This variant of the basic strategy could thus be called a longest step procedure. It is
better known however under the name optimum gradient method, ormethod ofsteepest
descent (for maximization, ascent). Theoretical investigations of convergence and rate
of convergence of the method can be found, e.g., in Akaike (1960), Goldstein (1962),
Ostrowski (1967), Forsythe (1968), Elkin (1968), Zangwill (1969), and Wolfe (1969, 1970,
1971). Zangwill proves convergence based on the assumptions that the line searches are
exact and the objective function is continuously twice di erentiable. Exactness of the one
dimensional minimization is not, however, a necessary assumption (Wolfe, 1969). It is
signi cant that one can only establish theoretically that a stationary point willbereached
(rF (x) = 0) or approached (krF (x)k < "" > 0). The stationary point is a minimum,
only if F (x) is convex and three times di erentiable (Akaike, 1960). Zellnik, Sondak, and
Davis (1962), however, show that saddle points are in practice an obstacle, only if the
search is started at one, or on a straight gradient trajectory passing through one. In other
cases numerical rounding errors ensure that the path to a saddle point is unstable.