Foundations of Data Science

B
A
Figure 8.1: Example where the natural clustering is not center-based.
to estimate densities of regions when data lies in high dimensions. So, as a preprocessing
step, one may want to first perform some type of projection into a low dimensional space,
such as SVD, before running a clustering algorithm.
We begin with a discussion of algorithms for center-based clustering, then examine
algorithms for high-density clusters, and then examine some algorithms that allow combining
the two. A resource for information on center-based clustering is the book chapter
[?].
8.2 k-means Clustering
We assume in this section that data points lie in R d and focus on the k-means criterion.
8.2.1 A maximum-likelihood motivation for k-means
We now consider a maximum-likelihood motivation for using the k-means criterion. Suppose
that the data was generated according to an equal weight mixture of k spherical
well-separated Gaussian densities centered at µ 1 , µ 2 , . . . , µ k , each with variance one in
every direction. Then the density of the mixture is
Prob(x) =
1 1
(2π) d/2 k
k∑
e −|x−µ i| 2 .
Denote by µ(x) the center nearest to x. Since the exponential function falls off fast, we
can approximate ∑ k
i=1 e−|x−µ i| 2 by e −|x−µ(x)|2 . Thus
Prob(x) ≈
i=1
1
(2π) d/2 k e−|x−µ(x)|2 .
The likelihood of drawing the sample of points x 1 , x 2 , . . . , x n from the mixture, if the
centers were µ 1 , µ 2 , . . . , µ k , is approximately
1 1
k n (2π) nd/2
n
∏
i=1
e −|x(i) −µ(x (i) )| 2 = ce − ∑ n
i=1 |x(i) −µ(x (i) )| 2 .
267