Foundations of Data Science

orthonormal basis w 1 , w 2 , . . . , w k of W so that w k is perpendicular to v 1 , v 2 , . . . , v k−1 .
Then
|Aw 1 | 2 + |Aw 2 | 2 + · · · + |Aw k−1 | 2 ≤ |Av 1 | 2 + |Av 2 | 2 + · · · + |Av k−1 | 2
since V k−1 is an optimal k − 1 dimensional subspace. Since w k is perpendicular to
v 1 , v 2 , . . . , v k−1 , by the definition of v k , |Aw k | 2 ≤ |Av k | 2 . Thus
|Aw 1 | 2 + |Aw 2 | 2 + · · · + |Aw k−1 | 2 + |Aw k | 2 ≤ |Av 1 | 2 + |Av 2 | 2 + · · · + |Av k−1 | 2 + |Av k | 2 ,
proving that V k is at least as good as W and hence is optimal.
Note that the n-dimensional vector Av i is a list of lengths (with signs) of the projections
of the rows of A onto v i . Think of |Av i | = σ i (A) as the “component” of the matrix
A along v i . For this interpretation to make sense, it should be true that adding up the
squares of the components of A along each of the v i gives the square of the “whole content
of the matrix A”. This is indeed the case and is the matrix analogy of decomposing a
vector into its components along orthogonal directions.
Consider one row, say a j , of A. Since v 1 , v 2 , . . . , v r span the space of all rows of A,
r∑
a j · v = 0 for all v perpendicular to v 1 , v 2 , . . . , v r . Thus, for each row a j , (a j · v i ) 2 =
|a j | 2 . Summing over all rows j,
n∑
n∑ r∑
|a j | 2 = (a j · v i ) 2 =
But
j=1
n∑ ∑
|a j | 2 = n
j=1
j=1
d∑
j=1 k=1
i=1
r∑
i=1
n∑
(a j · v i ) 2 =
j=1
r∑
|Av i | 2 =
i=1
i=1
r∑
σi 2 (A).
a 2 jk , the sum of squares of all the entries of A. Thus, the sum of
squares of the singular values of A is indeed the square of the “whole content of A”, i.e.,
the sum of squares of all the entries. There is an important norm associated with this
quantity, the Frobenius norm of A, denoted ||A|| F defined as
i=1
√ ∑
||A|| F =
j,k
a 2 jk .
Lemma 3.2 For any matrix A, the sum of squares of the singular values equals the square
of the Frobenius norm. That is, ∑ σ 2 i (A) = ||A|| 2 F .
Proof: By the preceding discussion.
The vectors v 1 , v 2 , . . . , v r are called the right-singular vectors. The vectors Av i form
a fundamental set of vectors and we normalize them to length one by
u i = 1
σ i (A) Av i.
Later we will show that u 1 , u 2 , . . . , u r similarly maximize |u T A| when multiplied on the
left and are called the left-singular vectors. Clearly, the right-singular vectors are orthogonal
by definition. We will show later that the left-singular vectors are also orthogonal.
43