Foundations of Data Science

3.6 Left Singular Vectors
Theorem 3.7 The left singular vectors are pairwise orthogonal.
Proof: First we show that each u i , i ≥ 2 is orthogonal to u 1 . Suppose not, and for some
i ≥ 2, u T 1 u i ≠ 0. Without loss of generality assume that u T 1 u i = δ > 0. (If u 1 T u i < 0
then just replace u i with −u i .) For ε > 0, let
Notice that v ′ 1 is a unit-length vector.
v ′ 1 = v 1 + εv i
|v 1 + εv i | .
Av 1 ′ = σ 1u 1 + εσ i u
√ i
1 + ε
2
has length at least as large as its component along u 1 which is
u T 1 ( σ 1u 1 + εσ i u
( )
√ i
) > (σ 1 + εσ i δ) 1 − ε2 > σ 1 + ε
2
2 1 − ε2 σ 2 1 + εσ i δ + ε3 σ 2 iδ > σ 1 ,
for sufficiently small ɛ, a contradiction to the definition of σ 1 . Thus u 1 · u i = 0 for i ≥ 2.
The proof for other u i and u j , j > i > 1 is similar. Suppose without loss of generality
that u T i u j > δ > 0.
( )
vi + εv j
A
= σ iu i + εσ j u
√ j
|v i + εv j | 1 + ε
2
has length at least as large as its component along u i which is
u T i ( σ 1u i + εσ j u
√ j
) > ( ) ( )
σ i + εσ j u T
1 + ε
2
i u j 1 − ε2 > σ
2 i − ε2 σ 2 i + εσ j δ − ε3 σ 2 iδ > σ i ,
for sufficiently small ɛ, a contradiction since v i + εv j is orthogonal to v 1 , v 2 , . . . , v i−1 and
σ i is defined to be the maximum of |Av| over such vectors.
Next we prove that A k is the best rank k, 2-norm approximation to A. We first show
that the square of the 2-norm of A − A k is the square of the (k + 1) st singular value of A.
This is essentially by definition of A k ; that is, A k represents the projections of the points
in A onto the space spanned by the top k singular vectors, and so A−A k is the remaining
portion of those points, whose top singular value will be σ k+1 .
Lemma 3.8 ‖A − A k ‖ 2 2 = σ2 k+1 .
∑
Proof: Let A = r ∑
σ i u i v T i be the singular value decomposition of A. Then A k = k T
σ i u i v i
and A − A k =
i=1
∑ r
i=k+1
σ i u i v i T . Let v be the top singular vector of A − A k . Express v as a
47
i=1