Evolution and Optimum Seeking

Chapter 2
Problems and Methods of
Optimization
2.1 General Statement of the Problems
According to whether one emphasizes the theoretical aspect (existence conditions of optimal
solutions) or the practical (procedures for reaching optima), optimization nowadays
is classi ed as a branch of applied or numerical mathematics, operations research, orof
computer-assisted systems (engineering) design. In fact many optimization methods are
based on principles which were developed in linear and non-linear algebra. Whereas for
equations, or systems of equations, the problem is to determine a quantity or set of quantities
such that functions which depend on them have speci ed values, in the case of an
optimization problem, an initially unknown extremal value is sought. Many of the current
methods of solution of systems of linear equations start with an approximation and successively
improve itby minimizing the deviation from the required value. For non-linear
equations and for incomplete or overdetermined systems this way of proceeding is actually
essential (Ortega and Rheinboldt, 1970). Thus many seemingly quite di erent and apparently
unrelated problems turn out, after a suitable reformulation, to be optimization
problems.
Into this class come, for example, the solution of di erential equations (boundary
and initial value problems) and eigenvalue problems, as well as problems of observational
calculus, adaptation, and approximation (Stiefel, 1965 Schwarz, Rutishauser, and Stiefel,
1968 Collatz and Wetterling, 1971). In the rst case, the basic problem again is to
solve equations in the second, the problem is often reduced to minimize deviations in the
Gaussian sense (sum of squares of residues) or the Tchebyche sense (maximum of the
absolute residues). Even game theory (Vogelsang, 1963) and pattern or shape recognition
as a branch of information theory (Andrews, 1972 Niemann, 1974) have features in
common with the theory of optimization. In one case, from among a stored set of idealized
types, a pattern will be sought that has the maximum similarity to the one presented in
another case, the search will be for optimal courses of action in con ict situations. Here,
two or more interests are competing. Each player tries to maximize his chanceofwinning
with regard to the way in which his opponent supposedly plays. Most optimization
5