download searchable PDF of Circuit Design book - IEEE Global ...

124 Gradient Optimization
function is known to be quadratic, as (5.9) for example, then y'''(x) in (5.28)
will be zero anyhow. The reader should understand this single-variable case
before proceeding. The multivariable case is formulated in exactly the same
way.
The multivariable function in (5.6) can be expanded by a Taylor series
about point p, where the displacement from p is
Ax=x-p. \ (5.29)
The vector p might be the location of the blind man standing at p=(I0, 10)T in
Figure 5.5. Then the Taylor series for a real function of the vector x is
F(Ax) = F(p) + g(p)TAx + !AxTH(p)Ax + h.o.t., (5.30)
where higher-order terms (h.o.t) are presumed to be insignificant. Matrix H is
known as the Hessian:
H~
a'F
ax;
il'F
ilx, ax,
a'F
ax,ax,
il'F
axj
(5.31 )
By differentiating (5.13), it is seen that H=A for a quadratic function. It is
thus possible to expand the quadratic sample function in (5.7) about an
arbitrary point, say p= (10, IO)T. The result in terms of (5.29) is
F(Ax) =292+ (100,28)Ax+ !Ax T [ _ ~~
-1~ ]AX,
(5.32)
where Ax, =x, - 10 and Ax, = x, - 10. This describes the function in Figure 5.5
with respect to point p. For quadratic functions, this is the same as shifting the
origin; the reader should replace x, by x, + 10 and x, by x, + 10 in (5.8) and
confirm that it is equivalent to (5.32).
Analogous to (5.13), the gradient of (5.30) is
VF(Ax)= g(p)+ H(p)Ax. (5.33)
So the blind man on a quadratic mountain at point p (Figure 5.5) could
calculate where the minimum should be with respect to that point. In a
manner similar to (5.19), the step to the minimum is
(5.34)
Note that the second derivatives in H must be known. For the central sample
function used as an example, the step from point p=(IO, IOl to the minimum
IS
Ax=-H-'g=_l(26 10)(-100)=(-5).
576 10 26 -28 -3
See Figure 5.5 to confirm this step.
(5.35)