Evolution and Optimum Seeking

78 Hill climbing Strategies
converges quadratically, i.e., after a nite number (maximum n) of steps the minimum of
a quadratic function is located. Myers (1968) and Huang (1970) show that, if the same
starting point ischosen and the objective function is of second order, the DFP algorithm
generates the same iteration points as the conjugate gradient method of Fletcher and
Reeves.
All these observations are based on the assumption that the computational operations,
including the line searches, are carried out exactly. Then H (k) always remains positivede
nite if H (0) was positive-de nite and the minimum search is stable, i.e., the objective
function is improved at each iteration. Numerical tests (e.g., Pearson, 1969 Tabak,
1969 Huang and Levy, 1970 Murtagh and Sargent, 1970 Himmelblau, 1972a,b), and
theoretical considerations (Bard, 1968 Dixon, 1972b) show that rounding errors and
especially inaccuracies in the one dimensional minimization frequently cause stability
problems the matrix H (k) can easily lose its positive-de niteness without this being due
to a singularity intheinverse Hessian matrix. The simplest remedy for a singular matrix
H (k) , or one of reduced rank, is to forget from time to time all the experience stored within
H (k) and to begin again with the unit matrix and simple gradient directions (Bard, 1968
McCormick and Pearson, 1969). To do so certainly increases the number of necessary
iterations, but in optimization as in other activities it is wise to put safety before speed.
Stewart (1967) makes use of this procedure. His algorithm is of very great practical
interest since he obtains the required information about the rst partial derivatives from
function values alone by means of a cleverly constructed di erence scheme.
3.2.3.2 Strategy of Stewart:
Derivative-free Variable Metric Method
Stewart (1967) focuses his attentiononchoosing the length of the trial step d (k)
i
approximation
g (k)
i ' Fx i x (k) = @F(x)
@xi x (k)
for the
to the rst partial derivatives in such away astominimize the in uence of rounding
errors on the actual iteration process. Two di erence schemes are available:
and
g (k)
i
g (k)
i = 1
d (k)
i
=
1
2 d (k)
i
h F (x (k) + d (k)
i ei) ; F (x (k) ) i
h F (x (k) + d (k)
i ei) ; F (x (k) ; d (k)
i ei) i
(forward di erence)
(central di erence) (3.32)
Application of the one sided (forward) di erence (Equation (3.32)) is preferred, since
it only involves one extra function evaluation. To simplify the computation, Stewart
introduces the vector h (k) ,whichcontains the diagonal elements of the matrix (H (k) ) ;1
representing information about the curvature of the objective function in the coordinate
directions.
The algorithm for determining the g (k)
i
i = 1(1)n runs as follows: