Numerical recipes

414 Chapter 10. Minimization or Maximization of Functions
Figure 10.5.1. Successive minimizations along coordinate directions in a long, narrow “valley” (shown
as contour lines). Unless the valley is optimally oriented, this method is extremely inefficient, taking
many tiny steps to get to the minimum, crossing and re-crossing the principal axis.
Conjugate Directions
This concept of “non-interfering” directions, more conventionally called conjugate
directions, is worth making mathematically explicit.
First, note that if we minimize a function along some direction u, then the
gradient of the function must be perpendicular to u at the line minimum; if not, then
there would still be a nonzero directional derivative along u.
Next take some particular point P as the origin of the coordinate system with
coordinates x. Then any function f can be approximated by its Taylor series
f(x) =f(P)+ � ∂f
xi +
∂xi i
1 � ∂
2
i,j
2f xixj + ···
∂xi∂xj
≈ c − b · x + 1
x · A · x
2
(10.5.1)
where
c ≡ f(P) b ≡−∇f| P [A]ij ≡ ∂2 �
f �
�
∂xi∂xj
�
P
(10.5.2)
The matrix A whose components are the second partial derivative matrix of the
function is called the Hessian matrix of the function at P.
y
start
x
Sample page from NUMERICAL RECIPES IN C: THE ART OF SCIENTIFIC COMPUTING (ISBN 0-521-43108-5)
Copyright (C) 1988-1992 by Cambridge University Press. Programs Copyright (C) 1988-1992 by Numerical Recipes Software.
Permission is granted for internet users to make one paper copy for their own personal use. Further reproduction, or any copying of machinereadable
files (including this one) to any server computer, is strictly prohibited. To order Numerical Recipes books or CDROMs, visit website
http://www.nr.com or call 1-800-872-7423 (North America only), or send email to directcustserv@cambridge.org (outside North America).