Foundations of Data Science

This decomposition of A can be viewed as analogous to writing a vector x in some
orthonormal basis v 1 , v 2 , . . . , v d . The coordinates of x = (x · v 1 , x · v 2 . . . , x · v d ) are the
projections of x onto the v i ’s. For SVD, this basis has the property that for any k, the
first k vectors of this basis produce the least possible total sum of squares error for that
value of k.
In addition to the singular value decomposition, there is an eigenvalue decomposition.
Let A be a square matrix. A vector v such that Av = λv is called an eigenvector and
λ the eigenvalue. When A satisfies a few additional conditions besides being square, the
eigenvectors are orthogonal and A can be expressed as A = V DV T where the eigenvectors
are the columns of V and D is a diagonal matrix with the corresponding eigenvalues on
its diagonal. If A is symmetric and has distinct singular values, then the singular vectors
of A are the eigenvectors. If a singular value has multiplicity d greater than one, the
corresponding singular vectors span a subspace of dimension d and any orthogonal basis
of the subspace can be used as the eigenvectors or singular vectors. 5
The singular value decomposition is defined for all matrices, whereas the more familiar
eigenvector decomposition requires that the matrix A be square and certain other
conditions on the matrix to ensure orthogonality of the eigenvectors. In contrast, the
columns of V in the singular value decomposition, called the right-singular vectors of A,
always form an orthogonal set with no assumptions on A. The columns of U are called
the left-singular vectors and they also form an orthogonal set (see Section 3.6). A simple
consequence of the orthonormality is that for a square and invertible matrix A, the inverse
of A is V D −1 U T .
Eigenvalues and eignevectors satisfy Av = λv. We will show that singular values and
vectors satisfy a somewhat analogous relationship. Since Av i is a n × 1 matrix (vector),
the matrix A cannot act on it from the left. But A T , which is a d × n matrix, can act on
this vector. Indeed, we will show that
Av i = d ii u i and A T u i = d ii v i .
In words, A acting on v i produces a scalar multiple of u i and A T acting on u i produces
the same scalar multiple of v i . Note that A T Av i = d 2 iiv i . The i th singular vector of A is
the i th eigenvector of the square symmetric matrix A T A.
3.2 Preliminaries
Consider projecting a point a i = (a i1 , a i2 , . . . , a id ) onto a line through the origin. Then
a 2 i1 + a 2 i2 + · · · + a 2 id = (length of projection) 2 + (distance of point to line) 2 .
5 When d = 1 there are actually two possible singular vectors, one the negative of the other. The
subspace spanned is unique.
39