Nonparametric Bayesian Discrete Latent Variable Models for ...

3.2 MCMC Inference in Dirichlet Process Mixture Models
variables. This might result in faster mixing chains compared to the Pólya urn samplers
in some cases. The conditional for ci is
ci | xi, π1, . . . , πM, θ1, . . . , θM ∼
M
πkiδk(·) (3.44)
where the mixing proportions of the posterior are given by πki ∝ πkF (xi | θk).
Algorithm 8 Gibbs sampling for truncated DP
The state of the Markov chain consists of the component parameters θ1, . . . , θM,
mixing proportions π1, . . . , πM and indicator variables ci, . . . , cN.
Repeatedly sample:
for all i = 1, . . . , N do {indicator updates}
Assign ci to one of the M components with probability ∝ πkF (xi | θk)
end for
for all k = 1, . . . , M do {parameter updates}
Update θk by sampling from its posterior ∝ F (xi, ∀ci = k | θk)G0
end for
for all k = 1, . . . , M do {mixing proportion updates }
using eq. (3.43)
Sample the posterior breaking points v∗ k
Set πk = v∗ k−1 k l=1 v∗ l
end for
This algorithm does not require conjugacy, as we do not need to integrate over the
parameters for indicator variable updates. Gibbs sampling is easy to implement for
the truncated Dirichlet process using the stick-breaking construction. However, it is
desirable to avoid approximations and sample from the exact posterior distribution. In
the following sections, we describe methods by Papaspiliopoulos and Roberts (2005)
and Walker (2006) that use the stick-breaking representation without truncating the
process.
Retrospective Sampling
Retrospective sampling of Papaspiliopoulos and Roberts (2005) suggests allocating components
as needed instead of using truncation. Since the stick lengths sum up to 1,
prior assignment of the indicator variables is straightforward by starting with only a
few breaks of the stick (components) and breaking the stick more as a point in the
unbroken part is sampled, see Figure 3.8. The posterior distribution of the indicator
variables is given by the combination of the mixing proportions and the likelihood.
In this representation, the clusters are not exchangeable since the mixing proportions
differ. Therefore, we would need to sum over the infinitely many cases to obtain the
normalizing constant for the posterior. Retrospective sampling provides a way to avoid
evaluating this infinite sum by using a Metropolis-Hastings step.
k=1
35