Foundations of Data Science

This can be viewed as the variance of X, defined as the sum of the variances of all its
entries.
m∑ p∑
Var(X) = Var (x ij ) = ∑ E ( ( )
∑ ∑
xij) 2 − E (xij ) 2 1
= p k a 2
p
ikb 2 2 kj − ||AB|| 2 F .
i=1 j=1 ij
ij
k
We want to choose p k to minimize this quantity, and notice that we can ignore the ||AB|| 2 F
term since it doesn’t depend on the p k ’s at all. We can now simplify by exchanging the
order of summations to get
∑ ∑ 1
p k a 2
p
ikb 2 2 kj = ∑
ij k k
k
(
1 ∑
p k
i
a 2 ik
) ( ∑
j
b 2 kj
)
= ∑ k
k
1
p k
|A (:, k) | 2 |B (k, :) | 2 .
What is the best choice of p k to minimize this sum? It can be seen by calculus 29 that the
minimizing p k are proportional to |A(:, k)||B(k, :)|. In the important special case when
B = A T , pick columns of A with probabilities proportional to the squared length of the
columns. Even in the general case when B is not A T , doing so simplifies the bounds, so
we will use it. This sampling is called “length squared sampling”. If p k is proportional to
|A (:, k) | 2 , i.e, p k = |A(:,k)|2 , then
||A|| 2 F
E ( ) ∑
||AB − X|| 2 F = Var(X) ≤ ||A||
2
F |B (k, :) | 2 = ||A|| 2 F ||B|| 2 F .
To reduce the variance, we can do s independent trials. Each trial i, i = 1, 2, . . . , s
yields a matrix X i as in (7.1). We take 1 s
∑ s
i=1 X i as our estimate of AB. Since the
variance of a sum of independent random variables is the sum of variances, the variance
of 1 s
∑ s
i=1 X i is 1 s Var(X) and so is at most 1 s ||A||2 F ||B||2 F . Let k 1, . . . , k s be the k’s chosen
in each trial. Expanding this, gives:
1
s
s∑
X i = 1 s
i=1
k
( A (:, k1 ) B (k 1 , :)
+ A (:, k 2) B (k 2 , :)
+ · · · + A (:, k )
s) B (k s , :)
. (7.2)
p k1 p k2 p ks
We will find it convieneint to write this as the product of an m × s matrix with a s × p
matrix as follows: Let C be the m × s matrix consisting of the following columns which
are scaled versions of the chosen columns of A:
A(:, k 1 )
√ spk1
, A(:, k 2)
√ spk2
, . . . A(:, k s)
√ spks
.
Note that the scaling has a nice property (which the reader is asked to verify):
29 By taking derivatives, for any set of nonnegative numbers c k , ∑ k
to √ c k .
E ( CC T ) = AA T . (7.3)
250
c k
p k
is minimized with p k proportional