Foundations of Data Science

8.6.2 Gaussian Mixture Model
A second example is the Gaussian Mixture Model with k spherical Gaussians as components,
discussed in Section 3.6.2. We saw there that for two Gaussians, each of variance
one in each direction, data points are at distance O( √ d) from their correct centers and
if the separation between the centers is O(1), which is much smaller than O( √ d), an
approximate optimal solution could be misclassifying almost all points. We already saw
in Section 3.6.2, that SVD of the data matrix A helps in this case.
We will show that a natural and simple SVD-based approach called Spectral Clustering
helps solve not only these two examples, but a more general class of problems for which
there may or may not be a stochastic model. The result can be qualitatively described
by a simple statement when k, the number of clusters, is O(1): If there is some clustering
with cluster centroids separated by at least a constant times the “standard deviation”,
then, we can find a clustering close to this clustering. This should remind the reader of
the mnemonic “Means separated by 6 or some constant number of standard deviations”.
8.6.3 Standard Deviation without a stochastic model
First, how do we define mean and standard deviation for a clustering problem without
assuming a stochastic model of data? In a stochastic model, each cluster consists of independent
identically distributed points from a distribution, so the mean of the cluster is
just the mean of the distribution. Analogously, we can define the mean as the centroid
of the data points in a cluster, whether or not we have a stochastic model. If we had
a distribution, it would have a standard deviation in each direction, namely, the square
root of the mean squared distance from the mean of the distribution in that direction.
But the same definition also applies with no distribution or stochastic model. We give
the formal definitions after introducing some notation.
Notation: We will denote by A the data matrix which is n × d, with each row a data
point and k will denote the number of clusters. A k-clustering will be represented by a
n × d matrix C; row i of C is the center of the cluster that a i belongs to. So, C will have
k distinct rows.
The variance of the clustering C along the direction v, where v is a vector of length
1, is the mean squared distance of data points from their cluster centers in the direction
v, namely, it is
1
n∑ (
(ai − c i ) · v ) 2
.
n
i=1
The variance may differ from direction to direction, but we define the variance, denoted
σ 2 , of the clustering to be the maximum over all directions, namely,
σ 2 (C) = 1 n∑
n max
(
(ai − c i ) · v ) 2 1 =
v|=1
n ||A − C||2 2.
i=1
278