Foundations of Data Science

5.2 Markov Chain Monte Carlo
The Markov Chain Monte Carlo (MCMC) method is a technique for sampling a multivariate
probability distribution p(x), where x = (x 1 , x 2 , . . . , x d ). The MCMC method is
used to estimate the expected value of a function f(x)
E(f) = ∑ x
f(x)p(x).
If each x i can take on two or more values, then there are at least 2 d values for x, so an
explicit summation requires exponential time. Instead, one could draw a set of samples,
each sample x with probability p(x). Averaging f over these samples provides an estimate
of the sum.
To sample according to p(x), design a Markov Chain whose states correspond to the
possible values of x and whose stationary probability distribution is p(x). There are two
general techniques to design such a Markov Chain: the Metropolis-Hastings algorithm and
Gibbs sampling, which we will describe in the next section. The Fundamental Theorem of
Markov Chains, Theorem 5.2, states that the average of f over states seen in a sufficiently
long run is a good estimate of E(f). The harder task is to show that the number of steps
needed before the long-run average probabilities are close to the stationary distribution
grows polynomially in d, though the total number of states may grow exponentially in d.
This phenomenon known as rapid mixing happens for a number of interesting examples.
Section 5.4 presents a crucial tool used to show rapid mixing.
We used x ∈ R d to emphasize that distributions are multi-variate. From a Markov
chain perspective, each value x can take on is a state, i.e., a vertex of the graph on which
the random walk takes place. Henceforth, we will use the subscripts i, j, k, . . . to denote
states and will use p i instead of p(x 1 , x 2 , . . . , x d ) to denote the probability of the state
corresponding to a given set of values for the variables. Recall that in the Markov chain
terminology, vertices of the graph are called states.
Recall the notation that p t is the row vector of probabilities of the random walk being
at each state (vertex of the graph) at time t. So, p t has as many components as there are
states and its i th component, p ti , is the probability of being in state i at time t. Recall
the long-term t-step average is
a t = 1 t [p 0 + p 1 + · · · + p t−1 ] . (5.1)
∑
The expected value of the function f under the probability distribution p is E(f) =
i f ip i where f i is the value of f at state i. Our estimate of this quantity will be the
average value of f at the states seen in a t step walk. Call this estimate γ. Clearly, the
expected value of γ is
E(γ) = ∑ (
)
1
t∑
f i Prob (walk is in state i at time j) = ∑ f i a ti .
t
i
j=1
i
145