multivariate production systems optimization - Stanford University

The second step of the Greenstadt (1967) modification, replacing small eigenvalues
with infinity, is based on the objective function being insensitive to any small eigenvalues.
Since the objective function is insensitive to a parameter with a small eigenvalue, this
parameter should not be allowed to influence the step-direction. This is precisely what is
achieved by replacing the eigenvalue with infinity. Replacing an eigenvalue with infinity is
accomplished by replacing the reciprocal of the eigenvalue with zero when inverting the
Hessian.
Conceptually, the Greenstadt (1967) modification applied to Newton’s Method
should result in a robust algorithm that is rapidly convergent and not affected by insensitive
parameters; the caveat being that it is very expensive computationally. For information on
the performance of the Newton-Greenstadt algorithm applied to petroleum engineering
problems, see Barua, Horne, Greenstadt, and Lopez (1989).
6.3 Quasi-Newton Methods
Newton’s Method is quadratically convergent when starting with a good initial estimate.
However, it is very expensive since the Hessian must be built and solved every iteration,
particularly when the Hessian is built with finite difference approximations. The idea
behind Quasi-Newton methods is to compromise on the speed of convergence while saving
on the expense associated with building and solving the Hessian. Instead of building and
solving an exact Hessian every iteration, Quasi-Newton methods attempt to update an
approximation of the Hessian.
Quasi-Newton theory is based on multidimensional generalizations of the secant
method. The object is to build up secant information as the iterations proceed. Suppose at
the kth iteration, the Newton step {xk+1 - xk } causes a change in the gradient of {gk+1 -
gk }. Then the next Hessian approximation will satisfy the secant passing through these
two iterates if
xk+1 - xk = Hk+1 gk+1 - gk
71
(6.28)
This condition is termed the Quasi-Newton condition and is the design criteria for Quasi-
Newton methods. Notice the similarity between this condition and the Newton condition,
Equation 6.12. This condition forces the Hessian approximation to exactly match the
gradient of the function in the displacement direction, {xk+1 - xk }. For multiple