Foundations of Data Science

But for l ≠ l ′ , by hypothesis the cluster centers are at least 15kσ(C)/ε apart. This implies
that
For i, k /∈ M , i, k in different clusters in C
|v i − v k | ≥ |c i − c k | − |c i − v i | − |v k − c k | ≥ 9kσ(C) . (8.4)
2ε
We will show by induction on the number of iterations of Step 3 the invariant that at the
end of t iterations of Step 3, S consists of the union of k − t of the k C l \ M plus a subset
of M. In other words, but for elements of M, S is precisely the union of k − t clusters
of C. Clearly, this holds for t = 0. Suppose it holds for a t. Suppose in iteration t + 1
of Step 3 of the algorithm, we choose an i 0 /∈ M and say i 0 is in cluster l in C. Then by
(8.3) and (8.4), T will contain all points of C l \ M and will contain no points of C l ′ \ M
for any l ′ ≠ l. This proves the invariant still holds after the iteration. Also, the cluster
returned by the algorithm will agree with C l except possibly on M provided i 0 /∈ M.
Now by (8.2), |M| ≤ ε 2 n and since each |C l | ≥ εn, we have that |C l \ M| ≥ (ε − ε 2 )n.
If we have done less than k iterations of Step 3 and not yet peeled off C l , then there are
still (ε − ε 2 )n points of C l \ M left. So the probability that the next pick i 0 will be in M
is at most |M|/(ε − ε 2 )n ≤ ε/k by (8.2). So with probability at least 1 − ε all the k i 0 ’s
we pick are out of M and the theorem follows.
8.7 High-Density Clusters
We now turn from the assumption that clusters are center-based to the assumption
that clusters consist of high-density regions, separated by low-density moats such as in
Figure 8.1.
8.7.1 Single-linkage
One natural algorithm for clustering under the high-density assumption is called single
linkage. This algorithm begins with each point in its own cluster and then repeatedly
merges the two “closest” clusters into one, where the distance between two clusters is
defined as the minimum distance between points in each cluster. That is, d min (C, C ′ ) =
min x∈C,y∈C ′ d(x, y), and the algorithm merges the two clusters C and C ′ whose d min value
is smallest over all pairs of clusters breaking ties arbitrarily. It then continues until there
are only k clusters. This is called an agglomerative clustering algorithm because it begins
with many clusters and then starts merging, or agglomerating them together. 33 Singlelinkage
is equivalent to running Kruskal’s minimum-spanning-tree algorithm, but halting
when there are k trees remaining. The following theorem is fairly immediate.
33 Other agglomerative algorithms include complete linkage which merges the two clusters whose maximum
distance between points is smallest, and Ward’s algorithm described earlier that merges the two
clusters that cause the k-means cost to increase by the least.
281