Foundations of Data Science

First suppose x is in the row space V of R. From Lemma 7.6 P x = x, so for x ∈ V ,
(A − AP )x = 0. Since every vector can be written as a sum of a vector in V plus a vector
orthogonal to V , this implies that the maximum must therefore occur at some x ∈ V ⊥ .
For such x, by Lemma 7.6, (A−AP )x = Ax. Thus, the question becomes: for unit-length
x ∈ V ⊥ , how large can |Ax| 2 be? To analyze this, write:
|Ax| 2 = x T A T Ax = x T (A T A − R T R)x ≤ ||A T A − R T R|| 2 |x| 2 ≤ ||A T A − R T R|| 2 .
This implies that ||A − AP || 2 2 ≤ ||A T A − R T R|| 2 . So, it suffices to prove that ||A T A −
R T R|| 2 2 ≤ ||A|| 4 F /r which follows directly from Theorem 7.5, since we can think of RT R
as a way of estimating A T A by picking according to length-squared distribution columns
of A T , i.e., rows of A. This proves Proposition 7.7.
Proposition 7.8 is easy to see. By Lemma 7.6, P is the identity on the space V spanned
by the rows of R, and P x = 0 for x perpendicular to the rows of R. Thus ||P || 2 F is the
sum of its singular values squared which is at most r as claimed.
We now briefly look at the time needed to compute U. The only involved step in
computing U is to find (RR T ) −1 or do the SVD of RR T . But note that RR T is an r × r
matrix and since r is much smaller than n and m, this is fast.
Understanding the bound in Theorem 7.9: To better understand the bound in
Theorem 7.9 consider when it is meaningful and when it is not. First, choose parameters
s = Θ(1/ε 3 ) and r = Θ(1/ε 2 ) so that the bound becomes E(||A − CUR|| 2 2) ≤ ε||A|| 2 F .
Recall that ||A|| 2 F = ∑ i σ2 i (A), i.e., the sum of squares of all the singular values of A.
Also, for convenience scale A so that σ1(A) 2 = 1. Then
σ 2 1(A) = ||A|| 2 2 = 1 and E(||A − CUR|| 2 2) ≤ ε ∑ i
σ 2 i (A).
This, gives an intuitive sense of when the guarantee is good and when it is not. If the
top k singular values of A are all Ω(1) for k ≫ m 1/3 , so that ∑ i σ2 i (A) ≫ m 1/3 , then
the guarantee is only meaningful when ε = o(m −1/3 ), which is not interesting because it
requires s > m. On the other hand, if just the first few singular values of A are large
and the rest are quite small, e.g, A represents a collection of points that lie very close
to a low-dimensional pancake and in particular if ∑ i σ2 i (A) is a constant, then to be
meaningful the bound requires ε to be a small constant. In this case, the guarantee is
indeed meaningful because it implies that a constant number of rows and columns provides
a good 2-norm approximation to A.
7.4 Sketches of Documents
Suppose one wished to store all the web pages from the WWW. Since there are billions
of web pages, one might store just a sketch of each page where a sketch is a few hundred
256