multivariate production systems optimization - Stanford University

If used properly, Newton’s Method offers the fastest rate of convergence available.
However, this robust rate of convergence is achieved at the expense of the stability of the
algorithm. Some of the shortcomings of the method are
• A good initial starting point is required for convergence. A poor estimate
will result in divergence.
• If Newton’s Method does converge, it converges to the nearest stationary
point. No guarantee can be made that this is the desired stationary point.
It could be a local minimum, a maximum, or a saddlepoint.
• The Hessian matrix is subject to numerical instabilities during the matrix
solution.
• If the analytic derivatives are not available, the derivatives necessary to
construct the gradient and Hessian will have to be evaluated by very
costly finite difference approximations.
6.2 Modifications to Newton’s Method
In practice, the first modification made to Newton’s Method is to incorporate a onedimensional
line search to obtain an improved step-length. Based largely on the credo of
“Look before you leap,” the line search is performed on the objective function in the
direction indicated by p. The line search locates, or approximates the location of, the
minimum in this single dimension. The distance to the minimum in this single dimension is
taken as the step-length ρ. This minimum is taken as the new estimate of x by taking a step
of length ρ in the p direction (Equation 6.13).
The next level of modifications made to Newton’s Method involve altering the
eigenvalues of the Hessian. The ideal modification will converge rapidly, with speed
approaching that of Newton’s Method, and will always be descending downhill towards
the minimum. A direction of descent is guaranteed if the Hessian is positive definite,
meaning that all of the eigenvalues are positive. Three methods are discussed that modify
Newton’s Method to produce a positive definite Hessian. These are the method of steepest
descent, the Marquardt (1963) modification, and the Greenstadt (1967) modification. The
method of steepest descent is included primarily for illustrative purposes and is not
recommended for practical usage.
68