Foundations of Data Science

Lemma 3.10 (Analog of eigenvalues and eigenvectors)
Av i = σ i u i and A T u i = σ i v i .
Proof: The first equation is already known. For the second, note that from the SVD, we
get A T u i = ∑ j σ jv j u j T u i , where since the u j are orthonormal, all terms in the summation
are zero except for j = i.
3.7 Power Method for Computing the Singular Value Decomposition
Computing the singular value decomposition is an important branch of numerical
analysis in which there have been many sophisticated developments over a long period of
time. The reader is referred to numerical analysis texts for more details. Here we present
an “in-principle” method to establish that the approximate SVD of a matrix A can be
computed in polynomial time. The method we present, called the power method, is simple
and is in fact the conceptual starting point for many algorithms. Let A be a matrix whose
SVD is ∑ i σ iu i v T i . We wish to work with a matrix that is square and symmetric. Let
B = A T A. By direct multiplication, using the orthogonality of the u i ’s that was proved
in Theorem 3.7,
( ) ( )
∑ ∑
B = A T A = σ i v i u T i σ j u j vj
T
i
j
= ∑ i,j
σ i σ j v i (u T i · u j )v T j = ∑ i
σ 2 i v i v T i .
The matrix B is square and symmetric, and has the same left and right-singular vectors.
In particular, Bv j = ( ∑ i σ2 i v i vi T )v j = σj 2 v j , so v j is an eigenvector of B with eigenvalue
σj 2 . If A is itself square and symmetric, it will have the same right and left-singular vectors,
namely A = ∑ σ i v i v T i and computing B is unnecessary.
i
Now consider computing B 2 .
( ∑
B 2 = σi 2 v i vi
T
i
)
) ( ∑
j
σ 2 j v j v T j
= ∑ ij
σ 2 i σ 2 j v i (v i T v j )v j
T
When i ≠ j, the dot product v i T v j is zero by orthogonality. 8
computing the k th power of B, all the cross product terms are zero and
B k =
r∑
i=1
σ 2k
i v i v i T .
8 The “outer product” v i v j T is a matrix and is not zero even for i ≠ j.
∑
Thus, B 2 = r σi 4 v i v T i . In
i=1
49