Nonparametric Bayesian Discrete Latent Variable Models for ...

4 Indian Buffet Process Models
ciate parameters of the existing unique features of object i with some of the auxiliary
parameters and draw values from the prior G0( ˆ θj | ξ) for the rest of the auxiliary parameters.
Since each feature has a parameter associated with it, the particular order of
the columns is important, unlike the case in the previous section. There are 2 ˆ K possible
combinations for the ˆ K unique features for object i. We denote the part of the matrix
with active features that are not unique to object i with Z−K (i) and the auxiliary
features that are not associated with any object other than (possibly) object i with ˆ Zl
where the index l ∈ [1, . . . , 2 ˆ K ] denotes the possible feature settings. (Note that all
entries of ˆ Zl are zero except the ith row.) We evaluate the posterior probabilities of all
possible ˆ Zl and sample from this distribution to decide on which to include. The joint
posterior for the ith row of ˆ Zl will consist of the prior Bernoulli probabilities of setting
each feature in the ith row to 0 or 1 (with probability
the data given Z −K (i), θ −K (i), ˆ Zl and ˆ Θ,
P ( ˆ Zl | X, Z −K (i), Θ −K (i), ˆ Θ) ∝ P
α/ ˆ K
N+α/ ˆ ), and the probability of
K
ˆZl P X | ˆ Zl, Z−K (i), Θ−K (i), ˆ
Θ . (4.39)
The IBLF models have close correspondences with the DPM models which allows
infinite components in the mixture model. The unique features of an object can be
thought of as the singleton components in the DP, and the inactive features as the mixture
components that do not have any data associated with them. The sampling scheme
described in this section (summarized as Algorithm 12) is inspired from ”Algorithm 8”
of Neal (2000) for inference in the Dirichlet process models with non-conjugate priors
(described in Section 3.2.2, Algorithm 5). However, note that Neal’s algorithm for DP
models is exact, whereas the algorithm described here for the IBLF models uses an approximation.
The quality of approximation gets better with larger values of truncation,
however it should be noted that the stationary distribution will be different than the
actual posterior distribution due to the approximation.
4.2.3 Metropolis-Hastings Sampling
As discussed above, not having conjugacy is a problem only when Gibbs sampling for
the unique features of the data point being considered. Meeds et al. (2007) suggest using
Gibbs sampling for the features with m−i,k>0 and treating the unique features separately
using Metropolis-Hastings sampling, which results in the true posterior distribution. We
describe the algorithm below and give a summary in Algorithm 13.
The set of features with m−i,k = 0 contains the finitely many unique features of
data point i and the infinitely many features that do not belong to any of the data
points. Instead of calculating the posterior over the number of new features and sampling
directly from this distribution as in Section 4.2.1, we can propose the number of unique
features ˆ K and the set of parameters associated with them ˆ Θ = ˆ θ 1: ˆ K from a proposal
distribution Q( ˆ K, ˆ Θ) and consider this proposal with a Metropolis Hastings acceptance
86