Foundations of Data Science

A B A B
Figure 8.7: Illustration of the objection to the consistency axiom. Reducing distances
between points in a cluster may suggest that the cluster be split into two.
between clusters and increase it significantly until it becomes d max and in addition
αd max exceeds all other distances, the resulting clustering has just one cluster containing
all of the points.
3. The single linkage clustering algorithm with the distance r stopping condition, stop
when the inter-cluster distances are all at least r, satisfies richness and consistency;
but not scale invariance.
Proof: (1) Scale-invariance is easy to see. If one scales up all distances by a factor, then
at each point in the algorithm, the same pair of clusters will be closest. The argument
for consistency is more subtle. Since edges inside clusters of the final clustering can only
be decreased and since edges between clusters can only be increased, the edges that led
to merges between any two clusters are less than any edge between the final clusters.
Since the final number of clusters is fixed, these same edges will cause the same merges
unless the merge has already occurred due to some other edge that was inside a final
cluster having been shortened even more. No edge between two final clusters can cause a
merge before all the above edges have been considered. At this time the final number of
clusters has been reached and the process of merging has stopped. Parts (2) and (3) are
straightforward.
Note that one may question both the consistency axiom and the richness axiom. The
following are two possible objections to the consistency axiom. Consider the two clusters
in Figure 8.7. If one reduces the distance between points in cluster B, they might get an
arrangement that should be three clusters instead of two.
The other objection, which applies to both the consistency and the richness axioms,
is that they force many unrealizable distances to exist. For example, suppose the points
were in Euclidean d space and distances were Euclidean. Then, there are only nd degrees
of freedom. But the abstract distances used here have O(n 2 ) degrees of freedom since the
distances between the O(n 2 ) pairs of points can be specified arbitrarily. Unless d is about
n, the abstract distances are too general. The objection to richness is similar. If for n
points in Euclidean d space, the clusters are formed by hyper planes each cluster may be
a Voronoi cell or some other polytope, then as we saw in the theory of VC dimensions
Section ?? there are only ( n
d)
interesting hyper planes each defined by d of the n points.
If k clusters are defined by bisecting hyper planes of pairs of points, there are only n dk2
possible clustering’s rather than the 2 n demanded by richness. If d and k are significantly
292