Nonparametric Bayesian Discrete Latent Variable Models for ...

4.1 The Indian Buffet Process
number of occupied tables is limited by the number of customers. There is no such limit
in IBP since customers can choose many dishes and the resulting matrix Z potentially
has infinitely many non-zero columns. Nevertheless, the expected number of total dishes
sampled remains finite, since the mean number of new dishes a customer samples decreases
reciprocally. It was assumed that the ith customer chooses Poisson(α/i) number
of new dishes. We can use the additive property of the Poisson distribution to deduce
that the total number of dishes sampled (the number of non-zero columns of Z) follows
a Poisson(αHN) distribution. This means, the effective dimension of the binary matrix
is determined by the IBP parameter α and the number of customers.
The exchangeability of both the customers and the dishes has been established by focusing
on the equivalence classes of the generated matrices using the lof function. The
process starts with the first customer selecting Poisson(α) number of dishes. Making
use of exchangeability, any customer can be the first one to choose. With this argument,
we can see that the total number of dishes sampled by each customer follows a
Poisson(α) distribution. Therefore, for a matrix with N rows (N customers), the number
of non-zero entries in Z follows a Poisson(αN) distribution, which implies that the
IBP produces sparse matrices.
4.1.1 A Distribution on Infinite Binary Matrices
The probability distribution over [Z] defined by the IBP that is given in eq. (4.2) can be
derived as the limit of a distribution over finite size binary matrices when the number of
columns approach infinity (Griffiths and Ghahramani, 2005). This approach is similar
to deriving the equivalent distribution to the Dirichlet process mixtures by considering
a mixture model with infinitely many components (see Section 3.1.5). We start with
defining the distribution over a finite binary matrix Z with N rows (customers) and K
columns (dishes), then take the limit K → ∞. Recall that an entry zik = 1 means that
the ith customer has sampled the kth dish, or equivalently, the ith object has the kth
feature. The graphical representation of the hierarchical model is depicted in Figure 4.2.
Each entry on the kth column of the finite-dimensional matrix is assumed to have a
probability µk of being 1,
(zik | µk) ∼ Bernoulli(µk). (4.3)
We will refer to µk as the feature presence probability for column k. Putting an independent
beta prior on each µk,
µk ∼ Beta( α
, 1), (4.4)
K
the posterior distribution for µk given the previous i − 1 entries of the kth column is
also beta by conjugacy;
(µk | zjk, 1 ≤ j < i) ∼ Beta( α
K + m.