Foundations of Data Science

mean.
(ii) There is “no free lunch”. Since we only work on a small random sample and
not on the whole input matrix, our error bounds will not be good for certain matrices.
For example, if the input matrix is the identity, it is intuitively clear that picking a few
random columns will miss the other directions. Indeed, the initial error bounds we prove
using length squared sampling are useful only for “numerically low-rank matrices”, which
we define later. But there are important applications, for example, Principal Component
Analysis, where one has numerically low-rank input matrices and these techniques are
useful. There are more sophisticated and time-consuming sampling methods which have
error bounds which are good even for non-numerically-low-rank matrices.
To the Reader: Why aren’t (i) and (ii) mutually contradictory ?
7.3.1 Matrix Multiplication Using Sampling
Suppose A is an m×n matrix and B is an n×p matrix and the product AB is desired.
We show how to use sampling to get an approximate product faster than the traditional
multiplication. Let A (:, k) denote the k th column of A. A (:, k) is a m × 1 matrix. Let
B (k, :) be the k th row of B. B (k, :) is a 1 × n matrix. It is easy to see that
AB =
n∑
A (:, k)B (k, :) .
k=1
Note that for each value of k, A(:, k)B(k, :) is an m × p matrix each element of which is a
single product of elements of A and B. An obvious use of sampling suggests itself. Sample
some values for k and compute A (:, k) B (k, :) for the sampled k’s and use their suitably
scaled sum as the estimate of AB. It turns out that nonuniform sampling probabilities
are useful. Define a random variable z that takes on values in {1, 2, . . . , n}. Let p k denote
the probability that z assumes the value k. We will solve for a good choice of probabilities
later, but for now just consider the p k as nonnegative numbers that sum to one. Define
an associated random matrix variable that has value
X = 1 p k
A (:, k) B (k, :) (7.1)
with probability p k . Let E (X) denote the entry-wise expectation.
E (X) =
n∑
Prob(z = k) 1 A (:, k) B (k, :) =
p k
k=1
n∑
A (:, k)B (k, :) = AB.
k=1
This explains the scaling by 1
p k
in X. In particular, X is a matrix-valued random variable
each of whose components is correct in expectation. We will be interested in
E ( )
||AB − X|| 2 F .
249