A First Course in Linear Algebra, 2017a

248 R n
Theorem 4.150: Existence of Minimizers
Let⃗y ∈ R m and let A be an m × n matrix.
Choose⃗z ∈ W = im(A) given by⃗z = proj W (⃗y), andlet⃗x ∈ R n such that⃗z = A⃗x.
Then
1. ⃗y − A⃗x ∈ W ⊥
2. ‖⃗y − A⃗x‖ < ‖⃗y −⃗u‖ for all ⃗u ≠⃗z ∈ W
We note a simple but useful observation.
Lemma 4.151: Transpose and Dot Product
Let A be an m × n matrix. Then
A⃗x •⃗y =⃗x • A T ⃗y
Proof. This follows from the definitions:
A⃗x •⃗y = ∑
i, j
a ij x j y i = ∑x j a ji y i =⃗x • A T ⃗y
i, j
♠
The next corollary gives the technique of least squares.
Corollary 4.152: Least Squares and Normal Equation
A specific value of⃗x which solves the problem of Theorem 4.150 is obtained by solving the equation
A T A⃗x = A T ⃗y
Furthermore, there always exists a solution to this system of equations.
Proof. For ⃗x the minimizer of Theorem 4.150, (⃗y − A⃗x) • A⃗u = 0forall⃗u ∈ R n and from Lemma 4.151,
this is the same as saying
A T (⃗y − A⃗x) •⃗u = 0
for all u ∈ R n . This implies
A T ⃗y − A T A⃗x =⃗0.
Therefore, there is a solution to the equation of this corollary, and it solves the minimization problem of
Theorem 4.150.
♠
Note that ⃗x might not be unique but A⃗x, the closest point of A(R n ) to ⃗y is unique as was shown in the
above argument.
Consider the following example.