Nonparametric Bayesian Discrete Latent Variable Models for ...

4.1 The Indian Buffet Process
and Ghahramani (2005) show that the distribution over the equivalence classes of the
matrices generated by this model and the IBP are the same, which we briefly present
below. See the referred paper for details of the derivation.
Conditioned on µk, the rows of the matrix are assumed to be independent, therefore
the joint distribution over a column is
P (zk | µk) = (µk) m.,k (1 − µk) N−m.,k , (4.10)
where m.,k = N i=1 zik, is the number of 1s in column k, i.e. the number of objects that
share the kth feature. We can integrate over µk using the Dirichlet integral (B.11), and
express the joint probability of the column directly in terms of the hyperparameter α:
P (zk) = P (zk | µk)P (µk) dµk
Γ(α/K + 1)
=
= α
K
k=1
Γ(α/K) µα/K−1
k
µ m.,k
k (1 − µk) N−m.,k dµk
Γ(m.,k + α/K)Γ(N − m.,k + 1)
.
Γ(N + α/K + 1)
k=1
(4.11)
Assuming the columns to be independent, the distribution over the whole matrix becomes
K
K α Γ(m.,k + α/K)Γ(N − m.,k + 1)
P (Z) = P (zk) =
. (4.12)
K Γ(N + α/K + 1)
Considering the lof -equivalence classes of the matrices, the number of matrices Z defined
by the above generative model that map to the equivalent matrix [Z] is
K!
2N ,
−1
h=1 Kh!
which leads to the following distribution over [Z],
P ([Z]) =
K!
2 N −1
h=1 Kh!
K
k=1
α
K
Γ(m.,k + α/K)Γ(N − m.,k + 1)
. (4.13)
Γ(N + α/K + 1)
This is the distribution of the equivalence classes for the binary matrix with K < ∞
columns. The distribution over the matrix with infinitely many columns can be obtained
by separating the terms for non-zero columns (i.e., columns with m.,k > 0) and the zero
columns and simply taking the limit as the number of columns K → ∞. The limiting
distribution is found to be equal to eq. (4.2).
We repeat the argument of the previous section: even though Z has infinitely many
columns, its distribution favors sparsity. We can compute the expected number of nonzero
entries in Z for the finite case, and take the limit to obtain the expression for the
73