Nonparametric Bayesian Discrete Latent Variable Models for ...

B Mathematical Appendix
Multinomial distribution is the distribution of a random variable that can take a
countable number of different values. The conjugate prior for the multinomial distribution
is the Dirichlet distribution, which is the multivariate generalization of the Beta
distribution.
The marginal distribution of each πi is beta
It follows from the definition of the Dirichlet distribution and the additive property of
the gamma distribution that if (π1, . . . , πk) ∼ D(α1, . . . , αk) and r1, . . . , rl are integers
such that 0 < r1, . . . , rl = k, then
r1
D( π1,
1
r2
πi . . . ,
r1+1
rl
rl−1+1
r1
πi | α1, . . . , αk) ∼ D( α1,
1
r2
αi . . . ,
r1+1
rl
rl−1+1
In particular, the marginal distribution of each πi is Beta αi, ( k 1 αj)
− αi .
Scale of the Dirichlet distribution
The first and second moments of πj are given by;
E{πj} = αj
α , Var(πi, πj) = αj(α − αj)
α 2 (α + 1) ,
αi), (B.6)
where α = K
j=1 αj is referred to as the scale of the distribution. The mean of the
distribution does not depend on the scale of the parameters, however scale determines
the spread.
Posterior distribution
The posterior distribution of the πj given multinomially distributed data is:
p(π1, . . . , πk | x1, . . . , xn) = D(α1 + n1, . . . , αk + nk),
where nj is the number of occurrence of the j th event. The parameters of the Dirichlet
distribution can be interpreted as the pseudo observations; for larger scale, the distribution
will be more concentrated around the mean, thus the prior will have more effect
on the posterior.
We can marginalize out the multinomial parameters πj using the Dirichlet integral
(B.11) and express the probability of the observation directly in terms of the Dirichlet
parameters:
p(x1, . . . , xn | α1, . . . , αk) =
Γ(α)
Γ( k
l=1 αl + n)
k
j=1
Γ(αj + nj)
Γ(αj)
Thus, the conditional distribution for a new observation xn+1 given the previous observations
is:
p(xn+1 = j | x1, . . . , xn, α1, . . . , αk) = αj + nj
k
i=1 αi + n .
120