Nonparametric Bayesian Discrete Latent Variable Models for ...

3 Dirichlet Process Mixture Models
(Neal, 2003). Thus, the problem of sampling from an arbitrary distribution reduces to
sampling from uniform distributions.
In Section 3.2.2 we have described an algorithm by Walker and Damien (1998) that
uses auxiliary variables to limit the space of sampling in the Pólya urn representation.
The auxiliary variable in that algorithm is chosen such that it has uniform distribution
defined by the likelihood value. Therefore, given the auxiliary variable, sampling from
the posterior reduces to sampling from a truncated version of the prior.
In this section, we describe a similar idea applied to the stick-breaking construction
of the DP by Walker (2006) that results in an elegant algorithm which is widely applicable.
The parameters, the mixing proportions and the indicator variables are repeatedly
updated. We introduce the temporary slice variable s when updating the indicators,
and discard it after the indicator update.
The distribution of the auxiliary variable s is defined such that the joint prior of s
and ci is a two-dimensional uniform distribution. Conditioning on s, ci is uniformly
distributed on a limited part of the prior space. Combining this with the likelihood, we
have the conditional posterior of the ci.
Recall that the prior probability of assigning an observation to one of the components
is given by the mixing proportions π = {π1, . . . , π∞},
Multiplying with the likelihood, the posterior is,
P (ci|π) = πci . (3.48)
P (ci|π, xi, θ) ∝ πciF (xi|θci ). (3.49)
We introduce the auxiliary slice variable s such that the joint posterior of the indicator
variable and s is
P (ci, s|π, xi, θ) ∝ I{s < πci }F (xi|θci ). (3.50)
Thus, the distribution of s given π and ci is uniform:
(s|π, ci) ∼ U(0, πci ) = I{s < πci }π−1 ci
and the distribution of ci conditioned also on s is
F (xi|θci ) if s < πci ,
P (ci|s, π, xi, θ) ∝
0 otherwise.
(3.51)
(3.52)
That is, the probability of assigning ci to components with mixing proportions less than
the slice variable is 0. Therefore, we only need to consider assignment to one of the
components that have a larger stick length than the slice variable s. This will clearly
be a finite number, rather than the infinitely many components.
Using slice sampling, we only need to represent the mixing proportions and the parameters
of the K † components. We allocate new components only when needed. The
slice value is sampled uniformly between 0 and πci . Note that the stick lengths are not
38