Nonparametric Bayesian Discrete Latent Variable Models for ...

3.5 Discussion
3.5 Discussion
In this chapter, we have considered the DPM models for density estimation and clustering.
The DP has been introduced by Ferguson (1973) and has been extensively used
especially since the development of MCMC methods for inference. There are several
different approaches to define the DP. In the beginning of this chapter, we have summarized
some of these approaches to give an insight about the DP and its properties. The
different ways of defining the same distribution has lead to several different inference
algorithms for DPM models. We have outlined some of the MCMC algorithms developed
for inference on the DPM models. The list of algorithms that we described is not
exhaustive, but we believe that they give a good overview of the development of the
techniques.
We compared the DPMoG models with conjugate and conditionally conjugate base
distributions. We showed that for the inference algorithm we used, mixing time of the
samplers can be vastly improved for the conditionally conjugate model by integrating
out one of the parameters while conditioning on the other. Using this improved sampling
scheme, inference for the conditionally conjugate model is not always computationally
more expensive than the conjugate one. The relative mixing performance depends on
the data modeled. The modeling performance of the conditionally conjugate model
for density estimation is found to be always better than the fully conjugate one, the
difference being significant in higher dimensions. These results suggest that one does
not have to resort to using the conjugate base distribution when it does not represent
the prior beliefs.
We have introduced the DPMFA model for modeling high dimensional data that is
believed to have a low dimensional representation as mixture of Gaussians with constrained
covariance matrices. We have demonstrated the modeling performance of the
DPMFA on a challenging clustering problem. Although the resulting clustering is successful,
the incremental sampling algorithm used is not feasible for practical use of the
model on high dimensional data and should be improved.
Difference between the DPM models and mixtures with finite but unknown number
of components is that the DPM takes into account the possibility of a yet not observed
data point to be coming from an unrepresented component through the concentration
parameter of the DPM both during training and when the predictive probabilities are
calculated. Although the RJMCMC or BDMCMC give a distribution over the possible
number of components, once an iteration is complete and a number of components
is chosen, it is only those components that are used to explain the new data. The
comparison of the performance if these models remains to be an interesting question.
65