Evolution and Optimum Seeking

Core storage required 233
term \core" here, but at the time these tests were performed, it was so called.) All indirect
methods of quadratic optimization, which solve the linear equations for the extremal,
require storage of order O(n 2 ) for the matrix of coe cients. The same holds for quasi-
Newton methods, except that here the signi cant r^oleisplayed by the approximation to
the inverse Hessian matrices. Most strategies that perform line searches in other than
coordinate directions also require O(n 2 )words for the storage of n vectors, each withn
coe cients. An exception to this rule is the conjugate gradient method of Fletcher and
Reeves, which at each stage only needs to retain the latest generated direction vector
for the subsequent iteration. Of the direct search methods included in the tests, the coordinate
methods, the method of Hooke and Jeeves,andtheevolution strategies work
with only O(n) words of core storage. How important the formal storage requirement
of an optimization method can be is shown by the maximum number of variables for
the tested strategies in Table 6.2. The limiting values range from 75 to 4,000 under the
given conditions. There exist, of course, tricks such as segmentation for enabling larger
programs to be run on smaller machines the cost of the strategy should then take into
account, however, the extra cost in preparation time for an optimization. (Here again,
modern virtual storage techniques and the relative cheapness of memory chips make the
considerations above look rather old-fashioned.)
In the following Table 6.11, all the strategies compared are listed again, together with
the order of magnitude of their required computation time as obtained from the rst set of
tests (columns 1 and 2). The third column shows how the computation time would vary
if each function call performed O(n 2 ) rather than O(n) operations, as would occur for the
worst case of a general quadratic objective function. The fourth column gives the storage
requirement, again only as an order of magnitude, and the fth displays the product
of the time and storage requirements from the two previous columns. Judging by the
computation computation time alone, the variable metric strategy seems the best suited
for true quadratic problems. In the least favorable case, however, it is more expensive
than an indirect method and only faster in special cases. Problems having a very simple
structure (e.g., Problem 1.1) can be solved just as well by direct search methods the time
they take isatworst only a constant factor more than that of a second order method.
If the total cost is measured by the product of time and storage requirements, all those
strategies that store a two dimensional array of data, show up badly at least for problems
with many variables. Since the coordinate methods have shown unreliable convergence,
the method of Hooke and Jeeves and the evolution strategies remain as the least costly
optimization methods. Their cost does not exceed that of indirect methods. The product
of time and storage is not such a bad measure of the total cost in many computing centers
jobs have been, in fact, charged with the product of storage requested in K words and
the time in seconds of occupation of the central processing unit (K-core-sec).
A comparison of the two membered and multimembered evolution strategies seems
clearly to favor the simpler method. This is not surprising as several individuals in the
multimembered procedure have to nd their way towards the optimum. In nature, this
process runs in parallel. Already in the early 1970s, rst e orts towards constructing
multi-processor computers were undertaken (see Barnes et al., 1968 Miranker, 1971).