Nonparametric Bayesian Discrete Latent Variable Models for ...

3 Dirichlet Process Mixture Models
the components are not exchangeable in this representation. Therefore, caution should
be taken for mixing over the cluster labels to avoid clustering bias. Porteus et al. (2006)
discuss this in detail and introduce moves that permutes or swaps the labels to improve
mixing over the cluster labels.
3.3 Empirical Study on the Choice of the Base Distribution
In the previous section, we have described several different algorithms for doing inference
on the DPM models with both conjugate and non-conjugate base distribution. We have
seen that inference in the conjugate DPM models is relatively straightforward. For some
models, it is not possible to specify priors conjugate to the likelihood. The question is
whether one should use conjugate priors at all when they are available. It is known
that generally, conjugacy limits the flexibility of the models. Is the computational ease
worth the price of worse modeling performance? Or, is using conjugate models really
computationally cheaper than the non-conjugate alternatives, and how does the modeling
performance change with the choice of priors? In this section, we seek to empirically
address these questions using a DP mixture of Gaussians model (DPMoG). We choose
to use a DPMoG because it is one of the most widely used DPM models for which it is
possible to employ a conjugate and a conditionally conjugate base distribution.
In the following, we give model formulations for both a conjugate and a conditionallyconjugate
base distribution. For both prior specifications, we define hyperpriors on G0
for robustness. We refer to the models with the conjugate and the conditionally conjugate
base distributions in short as the conjugate model and the conditionally conjugate
model, respectively. After specifying the model structure, we will discuss in detail
how to do inference on both models. We will show that mixing performance of the
non-conjugate sampler can be improved substantially by exploiting the conditional conjugacy.
We will present experimental results comparing the modeling performance of
the two models and the computational cost of the samplers on several data sets.
3.3.1 The Dirichlet Process Gaussian Mixture Model
The finite Gaussian mixture model may be written as:
p(xi|θ1, . . . , θK) =
K
j=1
πjN (xi|µ j, S −1
j ) (3.53)
where θj = {µ j, Sj} is the set of parameters for component j, πj are the mixing proportions
(which must be positive and sum to one), µ j is the mean vector for component
j, and Sj is the component precision (inverse covariance matrix). Defining a joint prior
distribution G0 on the component parameters and introducing indicator variables, the
40