Nonparametric Bayesian Discrete Latent Variable Models for ...

1 Introduction
belief in the prior. The specification of the base distribution for the DP, which corresponds
to the prior on the component parameters for infinite mixture models, is often
guided by mathematical and practical convenience. For DPM models, the use of conjugate
priors makes the analysis much more tractable, however conjugate priors might fail
to represent the prior beliefs. Specifically for the DPMoG model, the conjugate priors
have some unappealing properties with prior dependencies between the mean and the
covariance. Empirical assessment of the trade off between modeling performance and
computational cost encountered when using conjugate priors for DPMoG is one of the
primary goals of this thesis. In Section 3.3 we compare the DPMoG model with a conjugate
and a conditionally-conjugate base distribution in terms of modeling performance
and computational feasibility. It is possible to integrate out some of the parameters in
the conditionally conjugate model, which vastly improves mixing. We show that this improvement
makes possible the practical use of the more flexible conditionally conjugate
model.
Mixtures of factor analyzers (MFA) is a mixture model that has Gaussian components
with constrained covariance matrices. Assuming that most of the information lies in
a lower dimensional space, high dimensional data can be efficiently modeled using this
reduced parametrization. In Section 3.4 we define the Dirichlet process mixtures of
factor analyzers (DPMFA) model and apply it to a challenging problem of clustering
neuronal data, known as spike sorting.
The second group of nonparametric models we consider is the latent feature models
with infinite number of binary features. The IBP is a distribution over infinite binary
sparse matrices that has many correspondences to the DP. Using IBP as a prior over the
latent features, we can define nonparametric latent feature models. Chapter 4 starts
with the different approaches for defining the distribution induced by the IBP, and
discusses the parallels to the DP. Section 4.2 describes different MCMC algorithms for
inference on IBP models. The sampling algorithms are compared in Section 4.3 to give
an intuition about their general performance. We demonstrate the modeling capability
of the IBP models for learning latent features of handwritten digits in Section 4.4.
The elimination by aspects (EBA) model is an interesting choice model in which the
latent variables are used to represent the features of the alternatives in the choice set
that lead to the choice probabilities. In Section 4.5, we define the EBA model with
IBP prior on the latent features and infer the choice probabilities by learning the latent
features using this model. We conclude the thesis with a discussion in Chapter 5.
The next chapter gives a brief overview of Bayesian modeling to introduce the terminology
and the notation that will be used in this thesis. Some mathematical formulas
and statistical definitions are given in Appendix B.
2