Nonparametric Bayesian Discrete Latent Variable Models for ...

4 Indian Buffet Process Models
sample more µ (k)s until ˜ K < K † and extend Z by adding the necessary number of
columns of zero, see Figure 4.6 for a pictorial explanation. In the appendix we show
that the stick lengths µ (k) for new features k > K † can be drawn iteratively from the
following distribution:
p(µ (k) | µ (k−1), Z:,.>k=0) ∝ µ α−1
(k) (1 − µ (k)) N exp(α N
i=1 1
i (1 − µ (k)) i )
I{0 ≤ µ (k) ≤ µ (k−1)},
(4.51)
which is obtained by conditioning on the fact that there are no non-zero entries beyond
the kth column, eq. (4.29). We can use ARS to draw samples from (4.51) since it is
log-concave in log µ (k). Finally, parameters for the new represented features are drawn
from the prior θk ∼ H.
Given s, we only need to update zik for each i = 1, . . . , N and k = 1, . . . , ˜ K. The
conditional probabilities are:
p(zik = 1 | s, µ (k), Z−ik, θ 1:K †) ∝ µ (k)
µ ∗ f(xi | zik = 1, Z−ik, θ 1:K †) (4.52)
The µ ∗ denominator is needed when different values of zik induces different values of µ ∗
(e.g. if zik = 1 prior to the update, and was the only non-zero entry in the last non-zero
column of Z).
is
For k = 1, . . . , K † −1, combining (4.42) and (4.43), the conditional probability of µ (k)
p(µ (k) | µ (k−1), Z) ∝µ m·,k−1
(k) (1 − µ (k)) N−m·,k
I{µ (k+1) ≤ µ (k) ≤ µ (k−1)} (4.53)
where m·,k = N
i=1 zik. For k = K † , in addition to taking into account the probability
of z :,K † = 0, we also have to take into account the probability that all columns of Z
beyond K † is zero. The resulting conditional probability of µ (K † ) is given by (4.51) with
k = K † . Drawing from both (4.53) and (4.51) can be done using ARS. The conditional
probability of θk for each k = 1, . . . , K † , is as given before, eq. (4.35).
The feature columns are sorted in a specific way in the stick-breaking representation,
therefore they are no longer exchangeable. This can result in poor mixing over the
features, similar to the DP case as pointed out by Porteus et al. (2006). This problem
becomes more apparent when one of the features with a large index happens to contribute
highly to the likelihood. In this case, this feature would persist, which would
require keeping all other unnecessary features in between the representative ones even
though they are not active. We can consider moves that facilitate mixing over the feature
indices and use Metropolis-Hastings sampling to evaluate the proposals similar to
the ones proposed for the DP case by Porteus et al. (2006). However, the additional
constraint of the stick lengths being in a strictly decreasing order complicates the formulation
of the moves. In the following section, we describe an alternative method to
facilitate mixing in the stick-breaking representation.
92