Foundations of Data Science

Minimizing the sum of squared distances to cluster centers finds the maximum likelihood
µ 1 , µ 2 , . . . , µ k . This motivates using the sum of distance squared to the cluster centers.
8.2.2 Structural properties of the k-means objective
Suppose we have already determined the clustering or the partitioning into C 1 , C 2 , . . . , C k .
What are the best centers for the clusters? The following lemma shows that the answer
is the centroids, the coordinate means, of the clusters.
Lemma 8.1 Let {a 1 , a 2 , . . . , a n } be a set of points. The sum of the squared distances of
the a i to any point x equals the sum of the squared distances to the centroid of the a i plus
n times the squared distance from x to the centroid. That is,
∑
|a i − x| 2 = ∑ |a i − c| 2 + n |c − x| 2
i
i
where c = 1 n
n∑
a i is the centroid of the set of points.
i=1
Proof:
∑
|a i − x| 2 = ∑
i
i
= ∑ i
|a i − c + c − x| 2
|a i − c| 2 + 2(c − x) · ∑
(a i − c) + n |c − x| 2
i
Since c is the centroid, ∑ i
(a i − c) = 0. Thus, ∑ i
|a i − x| 2 = ∑ i
|a i − c| 2 + n |c − x| 2
A corollary of Lemma 8.1 is that the centroid minimizes the sum of squared distances
since the first term, ∑ |a i − c| 2 , is a constant independent of x and setting x = c sets the
i
second term, n ‖c − x‖ 2 , to zero.
Corollary 8.2 Let {a 1 , a 2 , . . . , a n } be a set of points. The sum of squared
∑
distances of
the a i to a point x is minimized when x is the centroid, namely x = 1 a
n i .
8.2.3 Lloyd’s k-means clustering algorithm
Corollary 8.2 suggests the following natural strategy for k-means clustering, known as
Lloyd’s algorithm. Lloyd’s algorithm does not necessarily find a globally optimal solution
but will find a locally-optimal one. An important but unspecified step in the algorithm is
its initialization: how the starting k centers are chosen. We discuss this after discussing
the main algorithm.
i
268