Foundations of Data Science

8.6.4 Spectral Clustering Algorithm
Spectral Clustering - The Algorithm
1. Select a value for ɛ.
2. Find the top k right singular vectors of data matrix A and let V be the n × k matrix of
the top k right singular vectors.
3. Select a random row v i of V and form a cluster with all rows v j such that |v i −v j | ≤ 6kσ
ɛ
4. Repeat the above step k times.
Theorem 8.6 If in a k-clustering C, every pair of centers is separated by at least 15kσ(C)/ε
and every cluster has at least εn points in it, then with probability at least 1−ε, Spectral
Clustering finds a clustering C ′ which differs from C in at most ε 2 n points.
Before we prove the theorem, we show that the condition that every pair of cluster
centers is separated by 15kσ(C)/ε holds for the Stochastic Block Model and Gaussian
Mixture models discussed above for appropriate parameter settings. For this, we will
need the following theorem from Random Matrix Theory which we do not prove here.
Theorem 8.7 Suppose B is a n×d matrix with mutually independent, zero mean, random
entries with variance ν in O(ln n/n) that are well-behaved. If |b ij | ≤ 1 for all i and j or if
b ij are Gaussian random variables, they are well-behaved. The theorem works in greater
generality. Then, with high probability,
||B|| 2 ≤ c √ n + d √ ν.
Now for the stochastic block model with two communities of n/2 people each and p, q, α,
and β as above, we have E(a ij ) = c ij with B = A − C:
Setting ν = p, the theorem gives
E(b ij ) = 0 ; var(b ij ) = p(1 − p) or q(1 − q) ≤ p.
||A − C|| 2 ≤ c √ n √ p = c √ α =⇒ σ(C) ≤
√ α
√ n
.
From (8.1) inter-center separation is α−β √ n
. Thus, the condition of the theorem is satisfied
as long as √ α ∈ Ω(α − β), which is a reasonable assumption in the regime when α is at
least a large constant.
The proof for the Gaussian mixture model is similar. Suppose we have a mixture of k
Gaussians and A is a data matrix with n independent, identically distributed samples from
the mixture as its rows. The Gaussians need not be spherical. Let σmax be the maximum
standard deviation of any of the k Gaussians in any direction. We again consider C to
279