Evolution and Optimum Seeking

Multidimensional Strategies 65
evaluation (lattice search, factorial design), in which the task is to set up mathematical
models of physical or other processes. This territory is entered for example byG.E.P.Box
(1957), Box and Wilson (1951), Box andHunter (1957), Box and Behnken (1960), Box
and Draper (1969, 1987), Box et al. (1973), and Beveridge and Schechter (1970). It will
not be covered in any more detail here.
3.2.2 Gradient Strategies
The Gauss-Seidel strategy very straightforwardly uses only directions parallel to the coordinate
axes to successively improve the objective function value. All other direct search
methods strive toadvance more rapidly by taking steps in other directions. To doso
they exploit the knowledge about the topology of the objective function gleaned from
the successes and failures of previous iterations. Directions are viewed as most promising
in which the objective function decreases rapidly (for minimization) or increases rapidly
(for maximization). Southwell (1946), for example, improves the relaxation by choosing
the coordinate directions, not cyclically, but in order of the size of the local gradient in
them. If the restriction of parallel axes is removed, the local best direction is given by
the (negative) gradient vector
with
rF (x)=(Fx1(x)Fx2(x):::Fxn(x)) T
Fxi(x) = @F
(x) for all i = 1(1)n
@xi
at the point x (0) . All hill climbing procedures that orient their choice of search directions
v (0) according to the rst partial derivatives of the objective function are called gradient
strategies. They can be thought of as analogues of the total step procedure of Jacobi for
solving systems of linear equations (see Schwarz, Rutishauser, and Stiefel, 1968).
So great is the number of methods of this type which have been suggested or applied
up to the present, that merely to list them all would be di cult. The reason lies in
the fact that the gradient represents a local property of a function. To follow the path
of the gradient exactly would mean determining in general a curved trajectory in the ndimensional
space. This problem is only approximately soluble numerically and is more
di cult than the original optimization problem. With the help of analogue computers
continuous gradient methods have actually been implemented (Bekey and McGhee, 1964
Levine, 1964). They consider the trajectory x(t) as a function of time and obtain it as
the solution of a system of rst order di erential equations.
All the numerical variants of the gradient method di er in the lengths of the discrete
steps and thereby also with regard to how exactly they follow the gradient trajectory.
The iteration rule is generally
x (k+1) = x (k) ; s (k) rF (x(k) )
krF (x (k) )k
It assumes that the partial derivatives everywhere exist and are unique. If F (x) is continuously
di erentiable then the partial derivatives exist and F (x) iscontinuous.