Nonparametric Bayesian Discrete Latent Variable Models for ...

3.1 The Dirichlet Process
Eq. (3.21b) is the combined probability for assigning xi to one of the infinitely many
empty components. That is, the prior probability of assigning a data point to a component
that is associated with other data points is proportional to the number of data
points that have been already assigned to that component, and the probability of it
having its own component is proportional to α. This approach is in accordance with
the suggestion in Section 4 of (Kingman, 1975). Note the similarity of the above probabilities
to eq. (3.8) and the correspondence of the indicator variables in the CRP and
the infinite mixture model.
3.1.6 Properties of the Distribution
Different approaches for defining the DP summarized above reveal interesting properties
of the random measure and allow development of different inference algorithms for
models using DPs. Here, we summarize some properties of the DP that are important
for building DPM models and developing inference techniques.
The posterior distribution given samples θ1, . . . , θn from the process is again a DP.
This property makes inference for DP models easier to handle relative to most other
nonparametric models.
The Pólya urn scheme and the CRP show that it is possible to sample from the DP
without having to represent the full nonparametric distribution. Thus, inference on
DPM models is possible by only representing samples from the nonparametric distribution
rather than the infinite dimensional distribution itself.
Draws from a DP are discrete with probability 1. As a consequence, there is positive
probability of draws from a DP being identical. This property might lead to undesirable
posterior properties for some models (Petrone and Raftery, 1997) but can also be
exploited for defining powerful models such as the hierarchical Dirichlet process (HDP)
(Teh et al., 2006) that allow sharing statistical strength between groups of data.
The DP is defined by two parameters, namely the base distribution G0 and the
concentration parameter α. The mean of the distribution is G0, therefore G0 can be seen
as the prior distribution that we center our nonparametric model on. The concentration
parameter represents the ”strength of belief” on this prior distribution, similar to the
scale of the parameters for the Dirichlet distribution.
The concentration parameter α controls both the smoothness (or discreteness) of the
random distributions and the size of the neighborhood (or variability) of G about G0. In
the DPM models, the prior for selecting a new component to represent the ith data point
is proportional to α as given in eq. (3.21b). Thus, the value of α influences the number
of active components, that is, the number of components that have data assigned to
them. The prior expected number of active components for a set of n data points is
given by n α
N
i=1 α+i−1 ≈ α log( α + 1), (Antoniak, 1974).
For small α, the model will have a few active components drawn from the base distribution
to represent the data, and for large α there will be many active components,
thus the distribution of the parameters will be concentrated around G0, hence the name.
It is not clear what the less-informative value of α corresponds to since α → 0 results in
representing the data with a single component and α → ∞ expresses strong belief that
21