Springer - Read

384 Appendix B Statistical Complements
ˆθ (ˆθ0, ˆθ1) ′ satisfies the “normal equations”
X ′ X ˆθ X ′ y,
where X is the n×2 matrix X [1, x]. Since 0 ≤ S(θ) and S(θ) →∞as θ →∞,
the normal equations have at least one solution. If ˆθ (1) and ˆθ (2) are two solutions of
the normal equations, then a simple calculation shows that
ˆθ (1) − ˆθ (2)
′
X ′
X ˆθ (1) − ˆθ (2)
0,
i.e., that X ˆθ (1) X ˆθ (2) . The solution of the normal equations is unique if and only if
the matrix X ′ X is nonsingular. But the preceding calculations show that even if X ′ X
is singular, the vector ˆy X ˆθ of fitted values is the same for any solution ˆθ of the
normal equations.
The argument just given applies equally well to least squares estimation for the
general linear model. Given a set of data points
(xi1,xi2,...,xim,yi), i 1,...,nwith m ≤ n,
the least squares estimate, ˆθ ′ ˆθ1,...,ˆθm of θ (θ1,...,θm) ′ minimizes
S(θ)
n
(yi − θ1xi1 −···−θmxim) 2 y − θ1x (1) −···−θmx (m) 2 ,
i1
where y (y1,...,yn) ′ and x (j) (x1j,...,xnj ) ′ , j 1,...,m. As in the previous
special case, ˆθ satisfies the equations
X ′ X ˆθ X ′ y,
where X is the n × m matrix X x (1) ,...,x (m) . The solution of this equation is
unique if and only if X ′ X nonsingular, in which case
ˆθ (X ′ X) −1 X ′ y.
If X ′ X is singular, there are infinitely many solutions ˆθ, but the vector of fitted values
X ˆθ is the same for all of them.
Example B.1.1 To illustrate the general case, let us fit a quadratic function
to the data
y θ0 + θ1x + θ2x 2
x 0 1 2 3 4
y 1 0 3 5 8