Evolution and Optimum Seeking

166 Comparison of Direct Search Strategies for Parameter Optimization
was best from the point of view of its results. So long as no generally recognized quality
function of this kind exists, the question of which optimization method is optimal remains
unanswered.
6.2 Theoretical Results
Classical optimization theory is concerned with establishing necessary and su cient existence
criteria for maxima and minima. It provides systems of equations but no iterative
methods of nding their solutions. Not even Dantzig's simplex method (1966) for solving
linear programming problems can be regarded as a direct result of theory{theoretical considerations
of the linear problem only show that the extremum sought, except in special
cases, must always lie in a corner of the polyhedron de ned by the constraints. With n
variables and m constraints (together with n non-negativity conditions) the number of
corners or points of intersection of the hypersurfaces formed by the constraints is also
limited to a maximum of m+n
. Even the systematic inspection of all the points of
n
intersection would be a nite optimization method. But not all the points of intersection
are also within the allowed region (Saaty, 1955, 1963). Muller-Merbach (1971) gives
mn ; m + 2 as an upper bound to the number of feasible corner points. The simplex
method, which is a method of steepest ascent along the edges of the polyhedron only
traverses a tiny fraction of all the corners. Dantzig (1966) refers to empirical evidence
that the number of necessary iterations increases as n, thenumber of variables, if the
number of constraints m is constant, or as m if (n ; m) is not too small. Since, in
the least favorable case, between m and 2 m exchange operations must be performed on
the tableau of (m +1)(n + 1) coe cients, the average computation time increases as
O(m2 n). In so-called degenerate cases, however, the simplex method can also become
in nite. The repeated cycling through the same corners must then be broken by arule
for randomly choosing the iteration step (Dantzig). From a theoretical point of view the
ellipsoid method of Khachiyan (1979) and the interior point method of Karmarkar (1984)
do have the advantage of polynomial time consumption even in the worst case.
The question of niteness of iterative methodsisalsoacentral theme of non-linear
programming. In this case the solution can lie at any point on the border or interior
of the enclosed region. For the special case that the objective function and all the constraint
functions are convex and multiply di erentiable Kuhn and Tucker (1951) and John
(1948) have derived necessary and su cient conditions for extremal solutions. Most of
the iteration methods that have beendeveloped on this basis are designed for problems
with a quadratic objective function and linear constraints. Representative of quadratic
programming are, for example, the methods of Beale (1956) and Wolfe (1959a). They
make extensive use of the algorithm of the simplex method and thus belong, according to
Hadley (1969), to the class of neighboring extremal point methods. Other strategies can
moveinto the allowed region in the course of the iterations. As far as the constraints permit
they take the direction of the gradient of the objective function. They are therefore
known as gradient methods of non-linear programming (Kappler, 1967). As their name
may suggest, however, they are not suitable for all non-linear problems. Their convergence
can be proved at best for di erentiable quasi-convex programs (Kunzi, Krelle, and