Nonparametric Bayesian Discrete Latent Variable Models for ...

3 Dirichlet Process Mixture Models
π1 π2 π3 π4 5
π1 π2 π3 π4 5
π π
π π
π1 π2 π3 π4 π5 π6 π7 π8
Figure 3.8: Illustration of extending the DP representation retrospectively. The black horizontal
line is a stick of unit length. The blue vertical lines show the breaking points. The
length of each broken piece corresponds to a mixing proportion. Initially there
are only four components represented (top). To decide on the assignment of the
data point, a value is sampled uniformly from [0, 1], shown by the arrow (middle).
If the part that the arrow points at is not already represented, more pieces are
represented retrospectively by breaking the stick and sampling parameter values for
the new pieces until the arrow falls on a represented piece (bottom).
At each iteration, we denote the last component that has data assigned to it as K † .
We only represent the parameters and the mixing proportions of components with index
k ≤ K † . We can assign a data point either to one of the represented components or to
one of the rest. The posterior for the indicator variable ci is given as
with normalizing constant
P (ci = k | xi) ∝ πkF (xi | θk) for k ≤ K †
P (ci = k | xi) ∝ πkMi for k > K †
8
8
(3.45)
κ(K †
) = πkF (xi | θk) + (1 − πk)Mi, (3.46)
K †
k=1
where Mi is a constant controlling the probability of proposing an assignment to one of
the unrepresented components. Papaspiliopoulos and Roberts (2005) choose Mi such
that posterior probability of allocating i to a new component is greater than the prior.
With the choice of Mi(K † ) = max k≤K † F (xi | θk), the Metropolis-Hastings acceptance
36
K †
k=1