Foundations of Data Science

Lloyd’s algorithm:
Start with k centers.
Cluster each point with the center nearest to it.
Find the centroid of each cluster and replace the set of old centers with the centroids.
Repeat the above two steps until the centers converge (according to some criterion, such
as the k-means score no longer improving).
This algorithm always converges to a local minimum of the objective. To show convergence,
we argue that the sum of the squares of the distances of each point to its cluster
center always improves. Each iteration consists of two steps. First, consider the step
that finds the centroid of each cluster and replaces the old centers with the new centers.
By Corollary 8.2, this step improves the sum of internal cluster distances squared. The
second step reclusters by assigning each point to its nearest cluster center, which also
improves the internal cluster distances.
A problem that arises with some implementations of the k-means clustering algorithm
is that one or more of the clusters becomes empty and there is no center from which to
measure distance. A simple case where this occurs is illustrated in the following example.
You might think how you would modify the code to resolve this issue.
Example: Consider running the k-means clustering algorithm to find three clusters on
the following 1-dimension data set: {2,3,7,8} starting with centers {0,5,10}.
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
The center at 5 ends up with no items and there are only two clusters instead of the
desired three.
As noted above, Lloyd’s algorithm only finds a local optimum to the k-means objective
that might not be globally optimal. Consider, for example, Figure 8.2. Here data
lies in three dense clusters in R 2 : one centered at (0, 1), one centered at (0, −1) and
one centered at (3, 0). If we initialize with, say, one center at (0, 1) and two centers
near (3, 0), then the center at (0, 1) will move to near (0, 0) and capture the points near
(0, 1) and (0, −1), whereas the centers near (3, 0) will just stay there, splitting that cluster.
Because the initial centers can substantially influence the quality of the result, there
has been significant work on initialization strategies for Lloyd’s algorithm. One popular
269