Linear Algebra, Theory And Applications, 2012a

12.5. LEAST SQUARES 297
Lemma 12.5.1 Let V and W be finite dimensional inner product spaces and let A : V → W
be linear. For each y ∈ W there exists x ∈ V such that
|Ax − y| ≤|Ax 1 − y|
for all x 1 ∈ V. Also, x ∈ V is a solution to this minimization problem if and only if x is a
solution to the equation, A ∗ Ax = A ∗ y.
Proof: By Theorem 12.2.4 on Page 291 there exists a point, Ax 0 , in the finite dimensional
subspace, A (V ) , of W such that for all x ∈ V,|Ax − y| 2 ≥|Ax 0 − y| 2 . Also, from
this theorem, this happens if and only if Ax 0 − y is perpendicular to every Ax ∈ A (V ) .
Therefore, the solution is characterized by (Ax 0 − y, Ax) = 0 for all x ∈ V which is the
same as saying (A ∗ Ax 0 − A ∗ y, x) = 0 for all x ∈ V. In other words the solution is obtained
by solving A ∗ Ax 0 = A ∗ y for x 0 .
Consider the problem of finding the least squares regression line in statistics. Suppose
you have given points in the plane, {(x i ,y i )} n i=1
and you would like to find constants m
and b such that the line y = mx + b goes through all these points. Of course this will be
impossible in general. Therefore, try to find m, b such that you do the best you can to solve
the system
⎛ ⎞ ⎛ ⎞
y 1 x 1 1 ( )
⎜ . ⎟ ⎜
⎝
⎠ =
⎟ m
⎝ . . ⎠
b
y n x n 1
⎛ ⎞
2
( ) y 1
m ⎜
which is of the form y = Ax. In other words try to make
A −
b
⎝
⎟
. ⎠
as small
∣
y n
∣
as possible. According to what was just shown, it is desired to solve the following for m and
b.
⎛ ⎞
( ) y 1
A ∗ m
A = A ∗ ⎜
b
⎝
⎟
. ⎠ .
y n
Since A ∗ = A T in this case,
( ∑ n
∑i=1 x2 i n
i=1 x i
∑ n
i=1 x )(
i m
n b
Solving this system of equations for m and b,
)
=
( ∑ n
i=1 x iy i ∑n
i=1 y i
m = − (∑ n
i=1 x i)( ∑ n
i=1 y i)+( ∑ n
i=1 x iy i ) n
( ∑ n
i=1 x2 i ) n − (∑ n
i=1 x i) 2
and
b = − (∑ n
i=1 x i) ∑ n
i=1 x iy i +( ∑ n
i=1 y i) ∑ n
i=1 x2 i
( ∑ n
i=1 x2 i ) n − (∑ n
i=1 x i) 2 .
One could clearly do a least squares fit for curves of the form y = ax 2 + bx + c in the
same way. In this case you solve as well as possible for a, b, and c the system
using the same techniques.
⎛
⎜
⎝
⎞
x 2 1 x 1 1
⎛
.
.
⎟
. ⎠
x 2 n x n 1
⎝ a b
c
⎞ ⎛
⎠ ⎜
= ⎝
y 1
.
y n
⎞
⎟
⎠
)