Evolution and Optimum Seeking

74 Hill climbing Strategies
i = @2
@s 2(P (x + svi)) = ;2 (b ; c) Fa +(c ; a) Fb +(a ; b) Fc
(b ; c)(c ; a)(a ; b)
Powell uses this quantity i for all subsequent interpolations in the direction vi as a scale
for the second partial derivative of the objective function. He scales the directions vi,
which in his case are not normalized, by1= p i. This allows the possibility of subsequently
carrying out a simpli ed interpolation with only two argument values, x and x + si vi.
It is a worthwhile procedure, since each direction is used several times. The predicted
minimum, assuming that the second partial derivatives have value unity, is then
x 0 = x + 1
2 si ; 1
si
[F (x + si vi) ; F (x)] vi
For the trial step lengths si, Powell uses the empirical recursion formula
s (k)
q
(k) (k;1)
i =0:4 F (x ) ; F (x )
Because of the scaling, all the step lengths actually become the same. A more detailed
justi cation can be found in Ho mann and Hofmann (1970).
Contrary to most other optimization procedures, Powell's strategy is available as a
precise algorithm in a tested code (Powell, 1970f). As Fletcher (1965) reports, this method
of conjugate directions is superior for the case of a few variables both to the DSC method
and to a strategy of Smith, especially in the neighborhood of minima. For manyvariables,
however, the strict criterion for adopting a new direction more frequently causes the old
set of directions to be retained and the procedure then converges slowly. A problem which
had a singular Hessian matrix at the minimum made the DSC strategy look better. In
a later article, Fletcher (1972a) de nes a limit of n = 10 to 20, above whichthePowell
strategy should no longer be applied. This is con rmed by the test results presented
in Chapter 6. Zangwill (1967) combines the basic idea of Powell with relaxation steps
in order to avoid linear dependence of the search directions. Some results of Rhead
(1971) lead to the conclusion that Powell's improved concept is superior to Zangwill's.
Brent (1973) also presents a variant of the strategy without derivatives, derived from
Powell's basic idea, which is designed to prevent the occurrence of linear dependence of
the search directions without endangering the quadratic convergence. After every n +1
iterations the set of directions is replaced by an orthogonal set of vectors. So as not to
lose all the information, however, the unit vectors are not chosen. For quadratic objective
functions the new directions remain conjugate to each other. This procedure requires
O(n 3 ) computational operations to determine orthogonal eigenvectors. As, however, they
are only performed every O(n 2 ) line searches, the extra cost is O(n) per function call and
is thus of the same order as the cost of evaluating the objective function itself. Results of
tests by Brent con rm the usefulness of the strategy.
3.2.3 Newton Strategies
Newton strategies exploit the fact that, if a function can be di erentiated any number of
times, its value at the point x (k+1) can be represented by a series of terms constructed at