Foundations of Data Science

(0,1)
(3,0)
(0,-1)
Figure 8.2: A locally-optimal but globally-suboptimal k-means clustering.
strategy is called “farthest traversal”. Here, we begin by choosing one data point as initial
center c 1 (say, randomly), then pick the farthest data point from c 1 to use as c 2 , then
pick the farthest data point from {c 1 , c 2 } to use as c 3 , and so on. These are then used
as the initial centers. Notice that this will produce the correct solution in the example in
Figure 8.2.
Farthest traversal can unfortunately get fooled by a small number of outliers. To
address this, a smoother, probabilistic variation known as k-means++ instead weights
data points based on their distance from the previously chosen centers, specifically, proportional
to distance squared. Then it selects the next center probabilistically according
to these weights. This approach has the nice property that a small number of outliers
will not overly influence the algorithm so long as they are not too far away, in which case
perhaps they should be their own clusters anyway.
An alternative SVD-based method for initialization is described and analyzed in Section
8.6. Another approach is to run some other approximation algorithm for the k-means
problem, and then use its output as the starting point for Lloyd’s algorithm. Note that
applying Lloyd’s algorithm to the output of any other algorithm can only improve its
score.
8.2.4 Ward’s algorithm
Another popular heuristic for k-means clustering is Ward’s algorithm. Ward’s algorithm
begins with each datapoint in its own cluster, and then repeatedly merges pairs of clusters
until only k clusters remain. Specifically, Ward’s algorithm merges the two clusters
that minimize the immediate increase in k-means cost. That is, for a cluster C, define
cost(C) = ∑ a i ∈C d2 (a i , c), where c is the centroid of C. Then Ward’s algorithm merges
the pair (C, C ′ ) minimizing cost(C ∪ C ′ ) − cost(C) − cost(C ′ ). Thus, Ward’s algorithm
can be viewed as a greedy k-means algorithm.
270