Evolution and Optimum Seeking

172 Comparison of Direct Search Strategies for Parameter Optimization
For a quadratic function with a full matrix of coe cients, just to evaluate the expression
xT Axrequires O(n2 ) basic arithmetical operations. If the order of magnitude is denoted
by O(nf ) then, assuming f
computation time is given by:
1, for all the optimization methods considered so far the
T n 2+f
n 3
The advantage of having fewer function-independent operations in the Fletcher-Reeves
method, therefore, only makes itself felt if the number of variables is small and the time
for one function evaluation is short.
All the variants of the basic second order strategies mentioned here can be tted, with
similar assumptions, into the above scheme. Among these are (Broyden, 1972)
Modi ed and quasi-Newton methods
Methods of conjugate gradients and conjugate directions
Variable metric strategies, with their variations using correction matrices of rank
one
There is no optimization method that has a cost rising with less than the third power
of the number of variables. Even the indirect procedure, in which the equations for the
necessary conditions for an extremum are set up and solved by conventional methods,
does not a ord any basic reduction in the computational e ort. If the objective function
is quadratic, a system of n simultaneous linear equations is obtained. To solve for the
n unknowns the Gaussian elimination method requires 1
3 n3 basic operations (multiplications
and divisions). According to Zurmuhl (1965) all the other direct methods, meaning
here non-iterative methods, are more costly, except in special cases. Methods involving
a stepwise approach to the solution of systems of linear equations (relaxation methods)
require an in nite number of iterations to reach an absolutely exact result. They converge
linearly and correspond to rst order optimization strategies (single step or Gauss-Seidel
methods and total step or gradient methods see Schwarz, Rutishauser, and Stiefel, 1968).
Only the method of Hestenes and Stiefel (1952) converges after a nite number of calculation
steps, assuming that the calculations are exact. It is a conjugate gradient method
for solving systems of linear equations with a symmetrical, positive-de nite matrix of
coe cients.
The main concern here is with direct, i.e., derivative-free, search strategies for optimization.
Finiteness of the search in the quadratic case and greater than linear convergence
can only be proved for the Powell method of conjugate directions and for
the Davidon-Fletcher-Powell variable metric method, which Stewart reformulated as a
derivative-free quasi-Newton method. Of the coordinates strategy, at best it can be said
that it converges linearly. The same holds for the simple gradient methods. There are also
versions of them in which the partial derivatives are obtained numerically. Since various
comparison tests have shown them to be rather ine ective in highly non-linear situations,
none is considered here. No theoretically founded statements about convergence rates
and Q-properties are available for the other direct strategies. The rate of progress dened
by Rechenberg (1973) for the evolution strategy with adaptive step length control