Nonparametric Bayesian Discrete Latent Variable Models for ...

2 Nonparametric Bayesian Analysis
straightforward. It is possible to make the model more flexible by using a hierarchy and
defining hyper-priors on the priors of the parameters. However, the assumed form of
the distribution may be too restrictive such that the model fails to accurately represent
the data.
Mixture models allow modeling distributions that do not belong to a parametric family,
by representing the distribution in terms of K > 1 distributions of known simple
form. Mixture modeling is a strong tool of probabilistic models since arbitrary distributions
can be successfully modeled using simple distributions. Deciding on the number
of components in the model is generally considered as a model selection problem. It is
possible to treat the number of components, K, also as a parameter of the model and
infer it from the observations (Richardson and Green, 1997; Stephens, 2000).
An alternative to parametric models is the nonparametric models which are models
with (countably) infinitely many parameters. Nonparametric models achieve high flexibility
and robustness by defining the prior to be a nonparametric distribution from
a space of all possible distributions. All parameters in a model may be assumed to
have a nonparametric prior distribution or the nonparametric prior can be defined over
only a subset of the parameters, resulting in fully nonparametric and semiparametric
models, respectively. In this thesis, we refer to all models with a nonparametric part as
nonparametric models regardless of the presence (or absence) of other parameters with
parametric prior distributions.
In spite of having infinitely many parameters, inference in the nonparametric models
is possible since only a finite number of parameters need to be explicitly represented.
This brings close the models with finite but unknown number of components and the
nonparametric models. However, there are conceptual and practical differences between
the two. For instance, to learn the dimensionality of the parametric model we need to
move between different model dimensions (Green, 1995; Cappé et al., 2003). On the
other hand, the nonparametric model always has infinitely many parameters although
many of them do not need to be represented in the computations. As a consequence, the
posterior of the parametric model is of known finite dimension whereas the posterior of
a nonparametric model is also nonparametric, that is, it is also represented by infinitely
many parameters. We will focus on nonparametric models in this work. For more details
on Bayesian nonparametric methods, refer to for example (Ferguson et al., 1992; Dey
et al., 1998; Walker et al., 1999; MacEachern and Müller, 2000; Gosh and Ramamoorthi,
2003; Müller and Quintana, 2004; Rasmussen and Williams, 2006).
Approximate Inference Techniques
For some simple models, it is possible to calculate the posterior distribution of interest
analytically, however this is not the case generally. The specific form of prior for
which the integral over the parameters can be evaluated to get the marginal likelihood is
referred to as the conjugate prior for the likelihood. Often, the integrals we need to compute
to get the posterior distribution is not tractable, therefore we need approximation
techniques for inference.
Approximate inference methods include Laplace approximation, variational Bayes,
6