multivariate production systems optimization - Stanford University

The minimum of a function can occur only where the gradient vector vanishes and the
gradient vector vanishes only at a stationary point. To find the stationary point of the
quadratic approximation, we take the gradient of Q
∂Q
∂x
and equate this to zero. The resulting equation
= g + H p (6.10)
H p = - g (6.11)
can only be satisfied by a stationary point. To solve for the step-direction p that will lead to
the stationary point, the Hessian must be inverted
p = - H -1 g (6.12)
The step-direction p indicates the displacement from the point x that would give the
stationary point of the quadratic model. The Newton step is defined as the product (ρ p)
where ρ is the scalar step-length and p is the step-direction vector. We can update our
estimate of the point x by taking a Newton step
x ← x + ρ p (6.13)
Thus, the method takes a step of length ρ in the p direction. In unmodified Newton’s
Method, the length of the Newton step is implicitly taken as unity.
If the objective function is quadratic in form, then the quadratic approximation is
exact and Newton's Method will converge to the stationary point in a single iteration. In
the more likely event that the objective function is not quadratic in form, the higher-order
terms that were ignored in the Taylor series expansion will become negligible as Newton’s
Method begins to converge on the stationary point. As the higher-order terms go to zero,
the quadratic model will provide a very good approximation to the local surface and the
method will approach quadratic convergence.
A key insight to Newton’s Method can be obtained by investigating the eigensystem
of the Hessian matrix. By performing spectral decomposition of the Hessian
H = V Λ V T (6.14)
65