Nonparametric Bayesian Discrete Latent Variable Models for ...

3 Dirichlet Process Mixture Models
different models are depicted in Figure 3.14. This figure shows that the distribution of
the number of components used by the CCDP is much broader and centered on higher
values. The number of data points assigned to the components averaged over different
positions in the chain is depicted in Figure 3.15.
3.3.4 Conclusions
The Dirichlet process mixtures of Gaussians model is one of the most widely used DPM
models. We have presented and compared the conjugate and conditionally conjugate
hierarchical Dirichlet process Gaussian mixture models. We presented two new MCMC
schemes that improve the convergence and mixing properties of the MCMC algorithm
of Neal (2000) for the conditionally conjugate Dirichlet process mixture model. The
convergence and mixing properties of the samplers have been demonstrated on example
data sets. The modeling properties of the conjugate and the conditionally conjugate
model have been empirically compared. The predictive accuracy of the CCDP model
is found to be better than the CDP model for all data sets considered, the difference
being larger in high dimensions. It is also interesting to note that the CDP model tends
to use less components than the CCDP model.
In the light of the empirical results, we conclude that marginalizing over one of the
parameters by exploiting conditional conjugacy leads to considerably faster mixing in
the conditionally conjugate model. When using this trick, the fully conjugate model
is not necessarily computationally cheaper. The DP Gaussian mixture model with the
more flexible prior specification (conditionally conjugate prior) can be used on higher
dimensional density estimation problems, resulting in better density estimates than the
model with conjugate prior specification.
3.4 Dirichlet Process Mixtures of Factor Analyzers
Factor analysis (FA) is a well known latent variable model that models the correlation
structure in the data. The mixture of factor analyzers (MFA) model combines FA with
a mixture model, allowing each component to have a different latent representation.
The generative model for FA is given by x = Λz + µ + ε, where z is the hidden
factor, Λ the factor loading matrix, and ε the measurement noise. The factors and
noise are assumed to be Gaussian distributed, z ∼ N (0, I) and ε ∼ N (0, Ψ) where Ψ is
a diagonal matrix. Therefore, x is also Gaussian distributed with mean µ and covariance
ΛΛ T + Ψ. Considering a mixture of factor analyzers (MFA) with K components, the
data distribution becomes
p(x) =
K
j=1
πjN (µ j, ΛjΛj T + Ψ), (3.68)
where πj denote the mixing proportions. Each component has a separate mean parameter
µ j and a factor loading matrix Λj. The diagonal uniqueness matrix Ψ is common
for all components, capturing the measurement noise in the data.
52