Foundations of Data Science

that make up the document. So, a ij , the frequency of the i th term in the j th document
is the sum over all topics l of the fraction of document j which is on topic l times the
frequency of term i in topic l. In matrix notation,
Pictorially, we can represent this as:
A = BC.
⎛
T
E
R
M
⎜
⎝
DOCUMENT
A
n × m
⎞
=
⎟
⎠
⎛
T
E
R
M
⎜
⎝
TOPIC
B
n × r
⎞
⎟
⎠
⎛
T
O
P
⎜
I ⎝
C
DOCUMENT
C
r × m
⎞
⎟
⎠
This model is too simple to be realistic since the frequency of each term in the document
is unlikely to be exactly what is given by the equation A = BC. So, a more
sophisticated stochastic model is used in practice.
From the document collection we observe the n × m matrix A. Can we find B and
C such that A = BC? The top r singular vectors from a singular value decomposition
of A give a factorization BC. But there are additional constraints stemming from the
fact that frequencies of terms in one particular topic are nonnegative reals summing to
one and from the fact that the fraction of each topic a particular document is on are also
nonnegative reals summing to one. Altogether the constraints are:
1. A = BC and ∑ i
a ij = 1.
2. The entries of B and C are all non-negative.
3. Each column of B and each column of C sums to one.
Given the first two conditions, we can achieve the third by multiplying the i th column
of B by a positive real number and dividing the i th row of C by the same real number
without violating A = BC. By doing this, one may assume that each column of B sums
to one. Since ∑ i a ij is the total frequency of all terms in document j, ∑ i a ij = 1. Now
a ij = ∑ k b ikc kj implies ∑ i a ij = ∑ i,k b ikc kj = ∑ k c kj = 1. Thus, the columns of C also
sum to one.
The problem can be posed as one of factoring the given matrix A into the product of
two matrices with nonnegative entries called nonnegative matrix factorization.
300