Online Model Selection Based on the Variational Bayes

<strong>Online</strong> <strong>Model</strong> <strong>Selection</strong> <strong>Based</strong> on the Variational Bayes 1679
Appendix D
The VB method can be easily extended to the hierarchical Bayes model. Let
us consider the EFH model (see equation 2.1) with the prior distribution
P a (µ | ®0, c 0). The evidence for the hierarchical Bayes model is given by the
marginal likelihood with respect to the model parameter µ and the prior
hyperparameter ®0,
Z
P(XfTg) D dm (h )dm (®0)P(XfTg| µ )Pa (µ | ®0, c 0)P0(®0), (D.1)
where P0(®0) is the prior distribution for the prior hyperparameter ®0. The
free energy is de�ned by
Z
F(XfTg, Q) D dm (µ )dm (®0)dm (ZfTg)Q(µ, ®0, ZfTg)
�
P(XfTg, ZfTg| µ )Pa (µ
£ log
| ´
®0, c 0)P0(®0)
. (D.2)
Q(µ, ®0, ZfTg)
The hierarchical VB method can be obtained assuming the conjugate prior
for P a (µ | ®0, c 0),
P0(®0) D exp [b0 (a0®0c 0 ¡ Wa (®0, c 0)) ¡ Wa(a0, b0)] , (D.3)
and the factorization for the trial posterior distribution,
Q(µ, ®0, ZfTg) D Q h (µ )Q a (®0)Qz(ZfTg). (D.4)
The remaining calculations can be done by the same way as in the VB
method. The VB algorithm in this case consists of three steps. The posterior
probability for the hidden variable P(z(t)|x(t), N µ ) is calculated in the VB
E-step by using the ensemble average of the parameters
N
µ D hµi ® . (D.5)
The posterior hyperparameter ® is calculated in the VB M-step,
c ® D Thr(x, z)i N µ C c 0h®0ia, (D.6)
c 0h®0ia D c 0
Z
dm (®0)Q a (®0)®0 D 1
b
@Wa
(a, b), (D.7)
@a
together withc D T C c 0. The posterior hyper-hyperparameter (a, b) is then
calculated:
a D hµi ® C a0, (D.8)