Online Model Selection Based on the Variational Bayes

<strong>Online</strong> <strong>Model</strong> <strong>Selection</strong> <strong>Based</strong> on the Variational Bayes 1653
2.3 Evidence and Free Energy. The evidence for a data set XfTg is de-
�ned by
Z
P(XfTg) D dm (µ )P(XfTg| µ )P0(µ ), (2.4)
where dm (µ ) denotes a measure on the model parameter space and P0(µ )
denotes a prior distribution for the model parameters. The integration over
model parameters in equation 2.4 penalizes complex models with more degrees
of freedom (Bishop, 1995). This integration, however, is often dif�cult
to perform. In order to evaluate this integration, the VB method introduces
a trial probability distribution Q(µ, ZfTg), which approximates the posterior
probability distribution over the model parameter µ and the hidden
variables ZfTg D fz(t)|t D 1, . . . , Tg:
P(µ, ZfTg|XfTg) D P(XfTg, ZfTg| µ )P0(µ )
, (2.5)
P(XfTg)
where the probability distribution for a complete data set (XfTg, ZfTg) is
given by
P(XfTg, ZfTg| µ ) D
TY
P(x(t), z(t)| µ ). (2.6)
tD1
The Kullback-Leibler (KL) divergence between the trial distribution Q(µ,
ZfTg) and the true posterior distribution P(µ, ZfTg|XfTg) is given by
Z
KL(Q||P) D dm (µ )dm (ZfTg) Q(µ, ZfTg)
� ´
Q(µ, ZfTg)
£ log
P(µ, ZfTg|XfTg)
D log P(XfTg) ¡ F(XfTg, Q), (2.7)
where the free energy F(XfTg, Q) is de�ned by
Z
F(XfTg, Q) D dm (µ )dm (ZfTg) Q(µ, ZfTg)
�
P(XfTg, ZfTg| µ )P0(µ
£ log
)
´
] . (2.8)
Q(µ, ZfTg)
Therefore, the true posterior distribution is obtained by maximizing the free
energy with respect to the trial distribution Q(µ, ZfTg). The maximum of
the free energy is equal to the log evidence:
log P(XfTg) D max F(XfTg, Q) ¸ F(XfTg, Q), (2.9)
Q
Equation 2.9 implies that the lower bound for the log evidence can be evaluated
by using some trial posterior distributions Q(µ, ZfTg).