Evolution and Optimum Seeking

Theoretical Results 171
tions. Similar results apply in the case of the variable metric strategy, except that there
are an additional O(n 2 ) basic operations for matrix additions and multiplications. The
direct search method due to Powell evaluates neither rst nor second partial derivatives.
After every n + 1 line searches the direction vectors are rede ned, which requires O(n 2 )
values to be assigned. But since each one dimensional optimization counts as an iteration
step, only O(n) direct operations are attributed to each iteration. A convenient summary
of the relationships is given in Table 6.1. For simplicity only the terms of highest order in
the number of parameters n are accounted for, without their coe cients of proportionality.
So far we have no scale for comparison of the di erent function evaluations with each
other. Fletcher (1972a) and others consider an evaluation of the Hessian matrix to be
equivalent toO(n) gradient determinations or O(n 2 ) objective function calls. This type
of scaling is valid whenever the partial derivatives cannot be obtained in analytic form
and provided as functions, but are calculated approximately as quotients of di erences
obtained by trial steps in the coordinate directions. In any case it ought to be about
right if the objective function is of higher than second order. Accordingly the following
weighting of the function evaluations can be introduced on the table:
F : rF : r 2 F ^ = n 0 : n 1 : n 2
Before anything can be said about the overall computation cost, or time, one must
know howmany operations are required for calculating a value of the objective function.
In general a function of n variables will entail a cost that rises at least linearly with n.
Table 6.1: Number of operations required by the most important
basic strategies to minimize a quadratic objective function
in terms of the number of variables n (only orders
of magnitude)
Number of Number of operations per iteration
Strategy iterations
Function evaluations
F rF r
Elementary
2F operations
Newton
e.g., Newton-Raphson
Variable metric
e.g., Davidon
Conjugate gradients
e.g., Fletcher-Reeves
Conjugate directions
e.g., Powell
n 0 | n 0 n 0 n 3
n 1 n 0 n 0 | n 2
n 1 n 0 n 0 | n 1
n 2 n 0 | | n 1
n 0 n 1 n 2
Weighting factors