Evolution and Optimum Seeking

Multidimensional Strategies 75
another point x (k) :
where
F (x (k+1) )=F (x (k) )+h T rF (x (k) )+ 1
2 hT r 2 F (x (k) ) h + ::: (3.26)
h = x (k+1) ; x (k)
In this Taylor series, as it is called, all the terms of higher than second order are
zero if F (x) is quadratic. Di erentiating Equation (3.26) with respect to h and setting
the derivative equal to zero, one obtains a condition for the stationary points of a second
order function:
or
rF (x (k+1) )=rF (x (k) )+r 2 F (x (k) )(x (k+1) ; x (k) )=0
x (k+1) = x (k) ; [r 2 F (x (k) )] ;1 rF (x (k) ) (3.27)
If F (x) is quadratic and r 2 F (x (0) )ispositive-de nite, Equation (3.27) yields the
solution x (1) in a single step from any starting point x (0) without needing a line search. If
Equation (3.27) is taken as the iteration rule in the general case it represents the extension
of the Newton-Raphson method to functions of several variables (Householder, 1953). It
is also sometimes called a second order gradient method with the choice of direction and
step length (Crockett and Cherno , 1955)
v (k) = ;[r 2 F (x (k) )] ;1 rF (x (k) )
s (k) = 1 (3.28)
The real length of the iteration step is hidden in the non-normalized Newton direction
v (k) . Since no explicit value of the objective function is required, but only its derivatives,
the Newton-Raphson strategy is classi ed as an indirect or analytic optimization method.
Its ability to predict the minimum of a quadratic function in a single calculation at rst
sight looksvery attractive. This single step, however, requires a considerable e ort. Apart
from the necessityofevaluating n rst and n(n
+1) second partial derivatives, the Hessian
2
matrix r 2 F (x (k) )must be inverted. This corresponds to the problem of solving a system
of linear equations
r 2 F (x (k) ) 4 x (k) = ;rF (x (k) ) (3.29)
for the unknown quantities 4x (k) . All the standard methods of linear algebra, e.g., Gaussian
elimination (Brown and Dennis, 1968 Brown, 1969) and the matrix decomposition
method of Cholesky (Wilkinson, 1965), need O(n 3 ) computational operations for this
(see Schwarz, Rutishauser, and Stiefel, 1968). For the same cost, the strategies of conjugate
directions and conjugate gradients can execute O(n) steps. Thus, in principle, the
Newton-Raphson iteration o ers no advantage in the quadratic case.
If the objective function is not quadratic, then
v (0) does not in general point towards a minimum. The iteration rule (Equation
(3.27)) must be applied repeatedly.