Foundations of Data Science

⎡
⎢
⎣
A
m × n
⎤ ⎡
⎥ ⎢
⎦ ⎣
B
n × p
⎡
⎤
≈
⎥
⎦ ⎢
⎣
Sampled
Scaled
columns
of
A
m × s
⎤
⎡
⎣
⎥
⎦
Corresponding
scaled rows of B
s × p
⎤
⎦
Figure 7.3: Approximate Matrix Multiplication using sampling
Define R to be the s×p matrix with the corresponding rows of B similarly scaled, namely,
R has rows
B(k 1 , :)
√ , B(k 2, :)
√ , . . . B(k s, :)
√ .
spk1 spk2 spks
The reader may verify that
E ( R T R ) = A T A. (7.4)
From (7.2), we see that 1 s
∑ s
i=1 X i = CR. This is represented in Figure 7.3. We summarize
our discussion in Theorem 7.5.
Theorem 7.5 Suppose A is an m × n matrix and B is an n × p matrix. The product
AB can be estimated by CR, where C is an m × s matrix consisting of s columns of A
picked according to length-squared distribution and scaled to satisfy (7.3) and R is the
s × p matrix consisting of the corresponding rows of B scaled to satisfy (7.4). The error
is bounded by:
E ( )
||AB − CR|| 2 ||A|| 2
F ≤
F ||B||2 F
.
s
Thus, to ensure E (||AB − CR|| 2 F ) ≤ ε2 ||A|| 2 F ||B||2 F , it suffices to make s ≥ 1/ε2 . If
ε ∈ Ω(1) (so s ∈ O(1)), then the multiplication CR can be carried out in time O(mp).
When is this error bound good and when is it not? Let’s focus on the case that B = A T
so we have just one matrix to consider. If A is the identity matrix, then the guarantee is
not very good. In this case, ||AA T || 2 n2
F = n, but the right-hand-side of the inequality is . s
So we would need s > n for the bound to be any better than approximating the product
with the zero matrix.
More generally, the trivial estimate of zero (all zero matrix) for AA T makes an error in
Frobenius norm of ||AA T || F . What s do we need to ensure that the error is at most this?
If σ 1 , σ 2 , . . . are the singular values of A, then the singular values of AA T are σ1, 2 σ2, 2 . . .
and
||AA T || 2 F = ∑ σt 4 and ||A|| 2 F = ∑ σt 2 .
t
t
251