Nonparametric Bayesian Discrete Latent Variable Models for ...

4.3 Comparing Performances of the Samplers
denote the distribution over a finite (K × D) part of A also as matrix Gaussian,
A | σA ∼ N (0, σ 2 AI), (4.61)
where 0 denotes a K × D matrix of zeros. We put an IBP(α) prior on the latent binary
feature matrix Z, and a gamma prior on the IBP parameter α ∼ G(1, 1), completing
the model.
We compare the mixing performance of the two slice samplers (on the strictly decreasing
weights, and on the semi-ordered weights) and Gibbs sampling described in the
previous section. We chose to apply the samplers to simple synthetic data sets so that
we can be assured of convergence to the true posterior and that mixing times can be
estimated reliably in a reasonable amount of computation time for all samplers. We also
chose to use a conjugate model since Gibbs sampling requires conjugacy. However, note
that our implementation of the two slice samplers did not make use of the conjugacy.
We generated 1, 2 and 3 dimensional data sets from the model with data variance
fixed at σ2 x = 1, varying values of the strength parameter α = 1, 2 and the latent feature
variance σ2 A = 1, 2, 4, 8. For each combination of parameters we produced five data sets
with 100 data points, in total 120 data sets. For all data sets, we fixed σ2 x and σ2 A to
the generating values and learned the feature matrix Z and α.
We are interested in how the samplers on the nonparametric part of the model perform
Therefore, we fix the σx and σA values and learn Z, A and α for all cases. For each
data set and each sampler, we repeated 5 runs of 15, 000 iterations. We used the
autocorrelation coefficients of the number of represented features K ‡ and α (with a
maximum lag of 2500) as measures of mixing time. We found that mixing in K ‡ is
slower than mixing in α for all data sets and all three samplers. We also found that in
this regime the autocorrelation times do not vary with dimensionality or with σ2 A . In
Figure 4.7 we show the autocorrelation times of α and K ‡ over all runs, all data sets,
and all three samplers. As expected, the slice sampler using the decreasing stick lengths
ordering was always slower than the semi-ordered one. Surprisingly, we found that the
semi-ordered slice sampler was just as fast as the Gibbs sampler which fully exploits
conjugacy. This is about as well as we would expect a more generally applicable nonconjugate
sampler to perform. This is a motivating result for using the semi-ordered
slice sampler for inference on complex non-conjugate IBLF models.
The first algorithm introduced for inference on the non-conjugate IBLF models is the
approximate Gibbs sampling algorithm (Algorithm 12). Even though efficient sampling
techniques that do not need to use an approximation have been developed, it is interesting
to have an insight about how the approximate method performs compared to the
non-approximate ones. We compare the modeling performance of Algorithm 12 to the
conjugate Gibbs sampling results using the model described above.
We generated a synthetic data set of 6 × 6 images described in Griffiths and Ghahramani
(2005). The input images are composed of a combination of (a subset of) four
latent features and zero-mean Gaussian noise with 0.5 standard deviation. We used
the Gibbs sampling for conjugate models and the approximate Gibbs sampling for nonconjugate
models to learn the latent structure. For the approximate scheme we used
97