Foundations of Data Science

A clustering algorithm is consistent if increasing the distance between points in different
clusters and reducing the distance between points in the same cluster does not change
the clusters produced by the clustering algorithm.
We now show that no clustering algorithm A can satisfy all three of scale invariance,
richness, and consistency. 34
Theorem 8.13 No algorithm A can satisfy all three of scale invariance, richness, and
consistency.
Proof: Let’s begin with the simple case of just two points, S = {a, b}. By richness there
must be some distance d(a, b) such that A produces two clusters and some other distance
d ′ (a, b) such that A produces one cluster. But this then violates scale-invariance.
Let us now turn to the general case of n points, S = {a 1 , . . . , a n }. By richness, there
must exist some distance function d such that A puts all of S into a single cluster, and
some other distance function d ′ such that A puts each point of S into its own cluster.
Let ɛ be the minimum distance between points in d, and let ∆ be the maximum distance
between points in d ′ . Define d ′′ = αd ′ for α = ɛ/∆; i.e., uniformly shrink distances in d ′
until they are all less than or equal to the minimum distance in d. By scale invariance,
A(d ′′ ) = A(d ′ ), so under d ′′ , A puts each point of S into its own cluster. However, note
that for each pair of points a i , a j we have d ′′ (a i , a j ) ≤ d(a i , a j ). This means we can reach
d ′′ from d by just reducing distances between points in the same cluster (since all points
are in the same cluster under d). So, the fact that A behaves differently on d ′′ and d
violates consistency.
8.12.2 Satisfying two of three
There exist natural clustering algorithms satisfying any two of the three axioms. For
example, different versions of the single linkage algorithm described in Section 8.7 satisfy
different two of the three conditions.
Theorem 8.14
1. The single linkage clustering algorithm with the k-cluster stopping condition (stop
when there are k clusters), satisfies scale-invariance and consistency. We do not
get richness since we only get clusterings with k clusters.
2. The single linkage clustering algorithm with scale α stopping condition satisfies scale
invariance and richness. The scale α stopping condition is to stop when the closest
pair of clusters is of distance greater than or equal to αd max where d max is the maximum
pair wise distance. Here we do not get consistency. If we select one distance
34 A technical point here: we do not allow d to have distance 0 between two distinct points of S. Else,
a simple algorithm that satisfies all three properties is simply “place two points into the same cluster if
they have distance 0, else place them into different clusters”.
291