Foundations of Data Science

A
n × d
=
U
n × r
D
r × r
V T
r × d
Figure 3.2: The SVD decomposition of an n × d matrix.
the singular values of A. For any matrix A, the sequence of singular values is unique and
if the singular values are all distinct, then the sequence of singular vectors is unique up
to signs. However, when some set of singular values are equal, the corresponding singular
vectors span some subspace. Any set of orthonormal vectors spanning this subspace can
be used as the singular vectors.
3.5 Best Rank-k Approximations
Let A be an n × d matrix and think of the rows of A as n points in d-dimensional
space. Let
r∑
A = σ i u i vi
T
be the SVD of A. For k ∈ {1, 2, . . . , r}, let
i=1
A k =
k∑
σ i u i vi
T
i=1
be the sum truncated after k terms. It is clear that A k has rank k. We show that A k
is the best rank k approximation to A, where, error is measured in the Frobenius norm.
Geometrically, this says that v 1 , . . . , v k define the k-dimensional space minimizing the
sum of squared distances of the points to the space. To see why, we need the following
lemma.
Lemma 3.5 The rows of A k are the projections of the rows of A onto the subspace V k
spanned by the first k singular vectors of A.
Proof: Let a be an arbitrary row vector. Since the v i are orthonormal, the projection
of the vector a onto V k is given by ∑ k
i=1 (a · v i)v i T . Thus, the matrix whose rows are
45