Foundations of Data Science

8.4 Finding Low-Error Clusterings
In the previous sections we saw algorithms for finding a local optimum to the k-means
clustering objective, for finding a global optimum to the k-means objective on the line,
and for finding a factor 2 approximation to the k-center objective. But what about
finding a clustering that is close to the correct answer, such as the true clustering of
proteins by function or a correct clustering of news articles by topic? For this we need
some assumption about the data and what the correct answer looks like. In the next two
sections we will see two different natural assumptions, and algorithms with guarantees
based on them.
8.5 Approximation Stability
Implicit in considering objectives like k-means, k-median, or k-center is the hope that
the optimal solution to the objective is a desirable clustering. Implicit in considering
algorithms that find near-optimal solutions is the hope that near-optimal solutions are
also desirable clusterings. Let’s now make this idea formal.
Let C = {C 1 , . . . , C k } and C ′ = {C 1, ′ . . . , C k ′ } be two different k-clusterings of some
data set A. A natural notion of the distance between these two clusterings is the fraction
of points that would have to be moved between clusters in C to make it match C ′ , where
by “match” we allow the indices to be permuted. Since C and C ′ are both partitions of
the set A, this is the same as the fraction of points that would have to be moved among
clusters in C ′ to make it match C. We can write this distance mathematically as:
dist(C, C ′ 1
) = min
σ n
k∑
|C i \ C σ(i)|,
′
where the minimum is over all permutations σ of {1, . . . , k}.
Given an objective Φ (such as k-means, k-median, etc), define C ∗ to be the clustering
that minimizes Φ. Define C T to be the “target” clustering we are aiming for, such
as correctly clustering documents by topic or correctly clustering protein sequences by
their function. For c ≥ 1 and ɛ > 0 we say that a data set satisfies (c, ɛ) approximationstability
with respect to objective Φ if every clustering C with Φ(C) ≤ c · Φ(C ∗ ) satisfies
dist(C, C T ) < ɛ. That is, it is sufficient to be within a factor c of optimal to the objective
Φ in order for the fraction of points clustered incorrectly to be less than ɛ.
What is interesting about approximation-stability is the following. The current best
polynomial-time approximation guarantee known for the k-means objective is roughly a
factor of nine, and for k-median it is roughly a factor 2.7; beating a factor 1 + 3/e for
k-means and 1 + 1/e for k-median are both NP-hard. Nonetheless, given data that satisfies
(1.1, ɛ) approximation-stability for the k-median objective, it turns out that so long
as ɛn is sufficiently small compared to the smallest cluster in C T , we can efficiently find
272
i=1