Foundations of Data Science

9 Topic Models, Hidden Markov Process, Graphical
Models, and Belief Propagation
In the chapter on learning and VC dimension, we saw many model-fitting problems.
There we were given labeled data and simple classes of functions: half-spaces, support
vector machines, etc. The problem was to fit the best model from a class of functions
to the data. Model fitting is of course more general and in this chapter we discuss some
useful models. These general models are often computationally infeasible, in the sense
that they do not admit provably efficient algorithms. Nevertheless, data often falls into
special cases of these models that can be solved efficiently.
9.1 Topic Models
A topic model is a model for representing a large collection of documents. Each document
is viewed as a combination of topics and each topic has a set of word frequencies. For
a collection of news articles over a period, the topics may be politics, sports, science, etc.
For the topic politics, the words like “president” and “election” may have high frequencies
and for the topic sports, words like “batter” and “goal” may have high frequencies. A
news item document may be 60% on politics and 40% on sports. The word frequencies in
the document will be a convex combination of word frequencies for the two topics, politics
and sports, with weights 0.6 and 0.4 respectively. We describe this more formally with
vectors and matrices.
Each document is viewed as a “bag of words”. We disregard the order and context in
which each term occurs in the document and instead only list the frequency of occurrences
of each term. Frequency is the number of occurrences of the term divided by the total
number of all terms in the document. Discarding context information may seem wasteful,
but this approach works well in practice and is widely used. Each document is an n-
dimensional vector where n is the total number of different terms in all the documents
in the collection. Each component of the vector is the frequency of a particular term in
the document. Terms are words or phrases. Not all words are chosen as terms; articles,
simple verbs, and pronouns like “a”, “is”, and “it” may be ignored. /newline
Represent the collection of documents by a n × m matrix A, called the term-document
matrix, with one column per document in the collection. The topic model hypothesizes
that there are r topics and each of the m documents is a combination of topics. The
number of topics r is usually much smaller than the number of terms n. So corresponding
to each document, there is a vector with r components telling us the fraction of the
document that is on each of the topics. In the example above, this vector will have 0.6
in the component for politics and 0.4 in the component for sports. Arrange these vectors
as the columns of a r × m matrix C, called the topic-document matrix. There is a third
matrix B which is n × r. Each column of B corresponds to a topic; each component of
the column gives the frequency of a term in that topic. In the simplest model, the term
frequencies in documents are exact combinations of term frequencies in the various topics
299