Foundations of Data Science

⎛
customers
⎜
⎝
factors
⎞ ⎛ ⎞
movies
⎛
A =
U ⎝
V
⎟ ⎜ ⎟
⎠ ⎝ ⎠
⎞
⎠
Figure 3.3: Customer-movie data
of an n×k matrix U describing the customers and a k ×d matrix V describing the movies.
Finding the best rank k approximation A k by SVD gives such a U and V . One twist is
that A may not be exactly equal to UV , in which case A − UV is treated as noise. Another
issue is that SVD gives a factorization with negative entries. Non-negative matrix
Factorization (NMF) is more appropriate in some contexts where we want to keep entries
non-negative. NMF is discussed in Chapter 8.13
In the above setting, A was available fully and we wished to find U and V to identify
the basic factors. However, in a case such as movie recommendations, each customer may
have seen only a small fraction of the movies, so it may be more natural to assume that we
are given just a few elements of A and wish to estimate A. If A was an arbitrary matrix
of size n × d, this would require Ω(nd) pieces of information and cannot be done with a
few entries. But again hypothesize that A was a small rank matrix with added noise. If
now we also assume that the given entries are randomly drawn according to some known
distribution, then there is a possibility that SVD can be used to estimate the whole of A.
This area is called collaborative filtering and one of its uses is to recommend movies or to
target an ad to a customer based on one or two purchases. We do not describe it here.
3.9.3 Clustering a Mixture of Spherical Gaussians
Clustering is the task of partitioning a set of points into k subsets or clusters where
each cluster consists of “nearby” points. Different definitions of the quality of a clustering
lead to different solutions. Clustering is an important area which we will study in detail in
Chapter ??. Here we will see how to solve a particular clustering problem using singular
value decomposition.
Mathematical formulations of clustering tend to have the property that finding the
highest quality solution to a given set of data is NP-hard. One way around this is to
assume stochastic models of input data and devise algorithms to cluster data generated by
54