Nonparametric Bayesian Discrete Latent Variable Models for ...

2 Nonparametric Bayesian Analysis
belief. In this case, one would need to use priors from a larger family. Using a prior
distribution from a more general family will typically result in the integral in eq. (2.2)
being intractable, hence increased computational complexity in posterior calculations.
Conditionally conjugate priors provide a richer family of distributions while retaining
some of the tractability.
Bayesian inference refers to obtaining the posterior distributions for the parameters
of interest and extracting information about these parameters from the posterior. After
updating our beliefs about the model parameters using the Bayes’ rule, we can use the
model to make predictions about new observations x∗ , or about any unobserved quantity
φ whose distribution depends on the parameters, using the predictive distribution,
P (φ | X) = P (φ | Ψ)P (Ψ | X) dΨ. (2.4)
We may be asked to give a single prediction value ˆ φ, referred to as a point estimate,
rather than the distribution of the predictions. In this case, we choose a value ˆ φ from
the predictive distribution that minimizes the expected loss for a given loss function
L( ˆ φ, φ),
E L( ˆ φ, φ)
= L( ˆ φ, φ) P (φ | X) dφ. (2.5)
The optimal choice ˆ φ is referred to as the Bayes estimate.
In density modeling, we want to model the generating density in the light of the
observations. The Kullback-Leibler (KL) divergence is a standard measure of the discrepancy
between two distributions. The difference of the estimated density Q(φ) to
the true generating density P (φ) is given as
DKL(P Q) =
P (φ) log
P (φ)
dφ. (2.6)
Q(φ)
Since the generating density P is fixed, maximizing the log-likelihood minimizes the KL
divergence.
Some other widely used loss functions are the squared error loss, L( ˆ φ, φ) = φ − ˆ φ 2
which is minimized by the posterior mean, the absolute error loss L( ˆ φ, φ) = φ − ˆ φ
minimized by the posterior median, and the zero-one loss,
minimized by the posterior mode.
L( ˆ φ, φ) = 1 if φ − ˆ φ > ε
L( ˆ φ, φ) = 0 otherwise,
Assumptions on the Model Structure
(2.7)
Above, we referred to the set of all unknown variables in a model as the parameters,
denoted by Ψ. Some of the parameters in a model may have physical interpretations and
therefore their values may be of interest, whereas some are merely necessary for building
4