Evolution and Optimum Seeking

84 Hill climbing Strategies
or determining the parameters of a mathematical model from experimental data (nonlinear
regression and curve tting). Such objective functions are easier to handle because
Newton directions can be constructed straight away without second partial derivatives,
as long as the Jacobian matrix of rst derivatives of each term of the objective function
is given. The oldest iteration procedure constructed on this basis is variously known as
the Gauss-Newton (Gauss, 1809) method, generalized least squares method, or Taylor
series method. It has all the advantages and disadvantages of the Newton-Raphson strategy.
Improvements on the basic procedure are given by Levenberg (1944) and Marquardt
(1963). Wolfe's secant method (Wolfe, 1959b see also Jeeves, 1958) is the forerunner of
many variants which do not require the Jacobian matrix to be speci ed at the start but
construct it in the course of the iterations. Further details will not be described here the
reader is referred to the specialist literature, again up to 1973:
Barnes, J.G.P. (1965)
Bauer, F.L. (1965)
Beale (1970)
Brown and Dennis (1972)
Broyden (1965, 1969, 1971)
Davies and Whitting (1972)
Dennis (1971, 1972)
Fletcher (1968, 1971)
Golub (1965)
Jarratt (1970)
Jones (1970)
Kowalik and Osborne (1968)
Morrison (1968)
Ortega and Rheinboldt (1970)
Osborne (1972)
Peckham (1970)
Powell (1965, 1966, 1968b, 1970d,e, 1972a)
Powell and MacDonald (1972)
Rabinowitz (1970)
Ross (1971)
Smith and Shanno (1971)
Spath (1967) (see also Silverman, 1969)
Stewart (1973)
Vitale and Taylor (1968)
Zeleznik (1968)
Brent (1973) gives further references. Peckham's strategy is perhaps of particular
interest. It represents a modi cation of the simplex method of Nelder and Mead (1965)
and Spendley (1969) and in tests it proves to be superior to Powell's strategy (1965)
with regard to the number of function calls. It should be mentioned here at least that
non-linear regression, where parameters of a model that enter the model in a non-linear
way (e.g., as exponents) have to be estimated, in general requires a global optimization