Foundations of Data Science

e made up of the means of the Gaussians. Now the theorem is satisfied by A − C with
ν = σ 2 max. For k ∈ O(1), it is easy to see that the hypothesis of Theorem 8.6 is satisfied
provided the means of the component Gaussians are separated by Ω(σmax).
The proof of the Theorem 8.6 relies on a crucial lemma, which is simple to prove.
Lemma 8.8 Suppose A is an n × d matrix and suppose V is obtained by projecting the
rows of A to the subspace of the first k right singular vectors of A. For any matrix C of
rank less than or equal to k
||V − C|| 2 F ≤ 8k||A − C|| 2 2.
Note: V is just one matrix. But it is close to every C, in the sense ||V − C|| 2 F ≤
8k||A − C|| 2 2. While this seems contradictory, the point of the lemma is that for C far
away from V , ||A − C|| 2 will be high.
Proof: Since the rank of (V − C) is less than or equal to 2k,
||V − C|| 2 F ≤ 2k||V − C|| 2 2 and
||V − C|| 2 ≤ ||V − A|| 2 + ||A − C|| 2 ≤ 2||A − C|| 2 .
The last inequality follows since V is the best rank k approximation in spectral norm and
C has rank at most k. The lemma follows.
Proof of Theorem 8.6: We use the lemma to argue that barring a few exceptions, most
ā i are at distance at most 3kσ(C)/ε to the corresponding c i . This will imply that for
most i, the point ā i will be close distance at most to most 6kσ(C)/ε to most other ā j in
its own cluster. Since we assumed the cluster centers of C are well separated this will
imply that for most i and j in different clusters, |v i − v j | ≥ 9kσ(C)/ε. This will enable
us to prove that the distance based clustering step, Step 3, of the algorithm works.
Define M to be the set of exceptions:
M = {i : |ab i − c i | ≥ 3kσ(C)/ε}.
Since ||V − C|| 2 F = ∑ i |v i − c i | 2 ≥ ∑ i∈M |v i − c i | 2 ≥ |M| 9k2 σ 2 (C)
, using the lemma we get:
ε 2
|M| 9k2 σ 2 (C)
ε 2 ≤ ||V − C|| 2 F ≤ 8knσ 2 (C) =⇒ |M| ≤ 8ε2 n
9k . (8.2)
For i, j /∈ M, i, j in the same cluster in C,
|v i − v j | ≤ |v i − c i | + |c i − v j | ≤ 6kσ(C) . (8.3)
ε
280