Nonparametric Bayesian Discrete Latent Variable Models for ...

µ
µ
µ
µ
µ
µ
(1)
(2)
(3)
(4)
(5)
(6)
π(5)
π(6)
π(4)
π(3)
π(2)
4.1 The Indian Buffet Process
Figure 4.4: Stick-breaking construction for the DP and IBP. The black line at top is the stick
of unit length. The vertical lines show the break point of the stick. The horizontal
blue lines show the length of the stick left after each iteration, which determines
the values for the feature presence probabilities µ (k). The red dotted lines show the
discarded pieces of the stick corresponding to the mixing proportions for the DP.
We refer to this representation as the stick-breaking construction due to its correspondence
to the standard stick-breaking construction for the DP (Sethuraman, 1994;
Ishwaran and James, 2001).
Let πk be the length of stick piece discarded at iteration k. We have,
Making a change of variables vk = 1 − νk,
k−1
πk = (1 − νk)µ (k−1) = (1 − νk)
vk i.i.d.
k−1
∼ Beta(1, α), πk = vk
l=1
π
(1)
νl. (4.26)
l=1
(1 − vl) (4.27)
giving the standard stick-breaking construction for DP, eq. (3.16).
In both constructions the weights of interest are given in terms of the stick lengths. In
DP, the weights πk are the lengths of the discarded pieces, while in IBP, the weights µ (k)
are the length of the stick we keep, see Figure 4.4. This difference leads to the different
properties of the weights: for DP, the stick lengths sum to a length of 1 and are not
strictly decreasing, but are in a size-biased order. In IBP the stick lengths may sum to
a value greater than 1 but are decreasing. For both, the weights decrease exponentially
quickly in expectation.
The particular ordering of the columns results in some interesting properties of the
distribution on Z. As stated earlier, µ (k) is the prior probability of any of the objects
having feature k. There is a positive probability for any data point to have any of
the infinitely many features. However, since the feature presence probabilities have a
strictly decreasing order with exponential rate, in expectation the infinite dimensional
Z matrix has only finitely many non-zero entries.
We can calculate the distribution of the number of non-zero entries to the right of
77