Theory of Statistics - George Mason University

f(x) ≈ f(x∗) + (x − x∗) T 1
gf x∗ +
0.3 Some Basics of Linear Algebra 805
2 (x − x∗) T Hf
x∗ (x − x∗), (0.3.53)
because gf x∗ = 0, we have a general method of finding a stationary point
for the function f(·), called Newton’s method. If x is an m-vector, gf(x) is an
m-vector and Hf(x) is an m × m matrix.
Newton’s method is to choose a starting point x (0) , then, for k = 0, 1, . . .,
to solve the linear systems
(k)
Hf x p (k+1) (k)
= −gf x
(0.3.54)
for p (k+1) , and then to update the point in the domain of f(·) by
x (k+1) = x (k) + p (k+1) . (0.3.55)
The two steps are repeated until there is essentially no change from one iteration
to the next. If f(·) is a quadratic function, the solution is obtained in
one iteration because equation (0.3.53) is exact. These two steps have a very
simple form for a function of one variable.
0.3.4.3 Linear Least Squares
In a least squares fit of a linear model
y = Xβ + ɛ, (0.3.56)
where y is an n-vector, X is an n×m matrix, and β is an m-vector, we replace
β by a variable b, define the residual vector
and minimize its Euclidean norm,
r = y − Xb, (0.3.57)
f(b) = r T r, (0.3.58)
with respect to the variable b. We can solve this optimization problem by
taking the derivative of this sum of squares and equating it to zero. Doing
this, we get
d(y − Xb) T (y − Xb)
db
which yields the normal equations
= d(yT y − 2b T X T y + b T X T Xb)
db
= −2X T y + 2X T Xb
= 0,
Theory of Statistics c○2000–2013 James E. Gentle
X T Xb = X T y. (0.3.59)