download searchable PDF of Circuit Design book - IEEE Global ...

I
___I
126 Gradient Optimization
x,
x,
Figure 5.11. The negative gradient and Newton
vectors of a quadratic function. (Reprinted with
permission of Macmillan Publishing Co., Inc. from
Introduction to Optimization Techniques by M. Aoki.
Copyright © 1971 by ~asanao Aoki.J
generalization more easy to follow. The reader should not miss this opportu:
nity to "see" what'differential calculus has to say about multidimensional
functions, Taylor series representations, and the idea of linearization in the
case of Newton's method. The concepts of single-variable functions were
stated so that this transition could be related to calculus that every engineer
should recall.
Newton's method describes a change in each component of the variable
space, which converges to a minimum in just one step for quadratic functions
(see Figure 5.11). The Newton vector, or step, can proceed to the minimum
(the origin, as shown in Figure 5.11) in just one step. But what if the function
F(x) is not quadratic? Also, what if second partial derivatives are not known
or inconvenient to compute? Might not a sequence of moves in the direction
of steepest descent (negative, gradient) lead to the minimum? In how many
steps? These are questions that will be considered next.
5.2. Conjugate Gradient Search
Gradient optimization methods assume the availability of partial derivatives.
Usually, finding first partial derivatives adds considerable complexity to the
programming task or slows program execution time. Second partial deri~atives
are even less' convenient to obtain. Fortunately, there are a number of search
methods that do not require second derivatives; the popular conjugate gradient
methods belong to this class. Methods that require only function values
without any derivatives will be mentioned briefly in Section 5.7.
Almost all optimization methods select a sequence of directions leading to a
minimum (or maximum) ftinction value. A minimum in any particular direction
is located by varying just one variable, usually some scalar that determines
the distance from the last "turning" point, and this procedure is called a
linear search.,The linear algebra jargon and the special case of linear searches
on quadratic surfaces will ,be described. Several elementary search direction