Evolution and Optimum Seeking

Multidimensional Strategies 69
devised a variant of the procedure (see also Sections 3.2.1.5 and 3.2.1.6). Lowe (1964)
has gathered together the various schemes of trial steps for the EVOP strategy. The
philosophy oftheEVOP strategy is treated in detail by Box and Draper (1969). Some
examples of applications are given by Kenworthy (1967). The e ciency of methods of
determining the gradient in the case of stochastic perturbations is dealt with by Mlynski
(1964a,b, 1966a,b), Sergiyevskiy and Ter-Saakov (1970), and others.
3.2.2.1 Strategy of Powell: Conjugate Directions
The most important idea for overcoming the convergence di culties of the gradient strategy
is due to Hestenes and Stiefel (1952), and again comes from the eld of linear algebra
(see also Ginsburg, 1963 Beckman, 1967). It trades under the names conjugate directions
or conjugate gradients. The directions fvi i = 1(1)ng are said to be conjugate with
respect to a positive de nite n n matrix A if (Hestenes, 1956)
v T
i Avj =0 for all i j = 1(1)ni 6= j
A further property of conjugate directions is their linear independence, i.e.,
nX
i=1
i vi =0
only holds if all the constants f i i =1(1)ng are zero. If A is replaced by the unit matrix,
A = I, thenthevi are mutually orthogonal. With A = r 2 F (x) (Hessian matrix) the
minimum of a quadratic function is obtained exactly in n line searches in the directions
vi. This is a factor two better than the gradient Partan method. For general non-linear
problems the convergence rate cannot be speci ed. As it is frequently assumed, however,
that many problems behave roughly quadratically near the optimum, it seems worthwhile
to use conjugate directions. The quadratic convergence of the search with conjugate
directions comes about because second order properties of the objective function are
taken into account. In this respect it is not, in fact, a rst order gradient method, but
a second order procedure. If all the n rst and n
(n +1) second partial derivatives are
2
available, the conjugate directions can be generated in one process corresponding to the
Gram-Schmidt orthogonalization (Kowalik and Osborne, 1968). It calls for expensive
matrix operations. Conjugate directions can, however, be constructed without knowledge
of the second derivatives: for example, from the changes in the gradient vector in the
course of the iterations (Fletcher and Reeves, 1964). Because of this implicit exploitation
of second order properties, conjugate directions has been classi ed as a gradient method.
The conjugate gradients method of Fletcher and Reeves consists of a sequence of line
searches with Hermitian interpolation (see Sect. 3.1.2.3.4). As a rst search direction v (0)
at the starting point x (0) , the simple gradient direction
v (0) = ;rF (x (0) )
is used. The recursion formula for the subsequent iterations is
v (k) = (k) v (k;1) ;rF (x (k) ) for all k = 1(1)n (3.25)