download searchable PDF of Circuit Design book - IEEE Global ...

134 Gradient Optimization
Hessian, A-I Consider the function in (5.30) and its gradient in (5.33) when
H=A. When this gradient is zero, dx in (5.34) corresponds to the location of
the minimum value. Substituting this in (5.30) yields
(5.50)
This is the amount by which F(p) exceeds its minimum value.
The most popular optimization algorithm is the Fletcher-Powell method,
which was first described by Davidon (1959). It is also known as the variable
metric method, and it is worthwhile to observe that the latter name comes
directly from (5.50). Davidon noted that the matrix A-I in (5.50) associates a
squared length to any gradient. Therefore, he considered the inverse Hessian
matrix for any nonlinear function as its metric or measure of standard length.
His optimization method starts with a guess for H- 1 , usually the unit matrix
U. This produces the steepest descent move according to (5.34). Following
each iteration, Davidon "updates~' the estimate of the inverse Hessian, so that
it is exact when a minimum'is finally found. In the interim, Davidon's metric
varies, thus the name. There is also some statistical significance to the inverse
Hessian for least-squares analysis (see Davidon, 1959).
Variable metric methods in N dimensions require the storage of N(N + 1)/2
elements of the symmetric, estimated inverse Hessian matrix; so they are not
considered here for personal computers, although such methods converge
rapidly near minima. There are many variable metric algorithms, but Dixon
(1971) showed that most of these, which belong to a very large class of
algorithms, would produce equivalent results if the linear searches were
absolutely accurate. Instead, another kind of conjugate gradient algorithm will
be described, because it requires only 3N storage registers; it converges
rapidly to good engineering accuracy, but lacks the ultimate convergence
properties of variable metric methods. It is the Fletcher-Reeves conjugate
gradient algorithm, which was originally suggested for very large problems
(e.g., 1000 variables) on large computers. It is very effective for many
problems (e.g., up to 25 variables) on desktop computers. The nature of the
conjugate gradient search direction is described next, followed by a description
of the Fletcher-Reeves algorithm.
5.2.4. Fletcher-Reeves Conjugate Gradient Search Directions. Two vectors,
x and y, are said to be orthogonal (perpendicular) if their inner product is
zero, I.e.,
xTy=O=XTUy, (5.51)
where the unit matrix has been introduced to emphasize the following concept.
The vectors are said to be conjugate if
x T Ay=O, (5.52)
where A is a positive-definite matrix. Conjugacy requires that the vec'tors are
not parallel. More remarkably, conjugate vectors relate to A-quadratic forms
as depicted in Figure 5.19. Just as illustrated for ellipsoids without cross terms,