Nonparametric Bayesian Discrete Latent Variable Models for ...

3.4 Dirichlet Process Mixtures of Factor Analyzers
FA is generally used as a dimensionality reduction technique by assuming that the
dimension of the latent factors z is much less than the data dimension, Q ≪ D. For
this case, a FA can be interpreted as a Gaussian with a constrained covariance matrix.
Assuming that most of the structure lies in a lower dimensional space, MFA can be used
to model high dimensional data as a mixture of Gaussians with less computational cost.
Analytical inference is not possible for the MFA models but maximum likelihood
methods such as expectation maximization (EM) can be used to make point estimates
for the parameters (Ghahramani and Hinton, 1996) or MCMC methods can be used
for obtaining the posterior distribution (Utsugi and Kumagai, 2001). These algorithms
are straightforward to apply for fixed latent dimension and a fixed number of components.
Deciding on the latent dimension and the number of components to be used
is an important modeling decision for MFA which is usually dealt with by using cross
validation. Alternatively, learning the latent dimension or the number of components
can be included in the inference.
Bayesian inference on the latent dimension of the FA model has been studied by Lopes
and West (2004) using reversible jump MCMC (RJMCMC).
Ghahramani and Beal (2000) formulate a variational method for learning the MFA
model. They form a hierarchical MFA model by placing priors on the component parameters
and hyperpriors on some of the hyperparameters and do inference on the model
using a variational approximation to the log marginal likelihood. The dimension of
hidden factors for each component is determined by automatic relevance determination
(ARD) (MacKay, 1994; Neal, 1996), and the number of components are determined by
birth-death moves. The drawback of this method is that it is very sensitive to random
initialization of the parameters and since it is an approximation, it does not guarantee
finding the exact solution.
Fokoué and Titterington (2003) present a method for inferring both the latent dimension
and the number of components in an MFA model using birth and death MCMC
(BDMCMC) of Stephens (2000) which is an alternative to RJMCMC. They treat both
the hidden dimension and the number of components as a parameter of the model and
do inference on both using BDMCMC.
For these approaches, the number of components can be seen as a parameter of the
model which is inferred from the data. The DP prior allows a nonparametric Bayesian
formulation of the MFA model, eliminating the need to do inference on the number of
components necessary to represent the data.
In this section, we introduce the Dirichlet process mixtures of factor analyzers model
(DPMFA), a FA model with a DP prior over the distribution of the model parameters.
Dirichlet Process Mixtures of Factor Analysers
We start with the distributional assumptions of (Ghahramani and Beal, 2000) for the
parametric MFA model, and form the Dirichlet process MFA model by taking K → ∞.
In detail, the prior for the means µ j is Gaussian,
µ j ∼ N (ξ, R −1 ), (3.69)
53