Online Model Selection Based on the Variational Bayes

<strong>Online</strong> <strong>Model</strong> <strong>Selection</strong> <strong>Based</strong> on the Variational Bayes 1657
appendix B)
� 1
c
´ �
(@F/@® ) V® ,® D
(@F/@c )
, V ® ,c
V T
® ,c ´
, Vc ,c
� ´
Thr(x, z)i N C c 0®0 ¡ (T C c 0)®
£ µ , (2.30)
T C c 0 ¡ c
where the Fisher information matrix V for the posterior parameter distribution
P a (µ | ®, c ) is given by
1
V ® ,® D
c 2
½ � ´ �
@ log Pa @ log Pa @® @® T
´¾
®
D
D (µ ¡ hµi ® )(µ ¡ hµi ® ) TE
® ,
V ® ,c D 1
½ � ´ � ´¾
@ log Pa @ log Pa c @® @c ®
D « (µ ¡ hµi ® )(g(µ ) ¡ hg(µ )i ® )¬
® ,
½ � ´ � ´¾
@ log Pa @ log Pa Vc ,c D
@c @c ®
D « (g(µ ) ¡ hg(µ )i ® )(g(µ ) ¡ hg(µ )i ® )¬
® ,
g(µ ) D ® ¢ µ ¡ Ã (µ ). (2.31)
Since the Fisher information matrix V is positive de�nite, the free energy
maximization condition, 2.29, leads to the VB M-step equations, 2.17 and
2.18. From equation 2.30, it is shown that the VB M-step solution is a maximum
of the free energy with respect to (®, c ), as in the VB-E step.
The VB algorithm is summarized as follows. First, c is set to (T C c 0).
In the VB E-step, the ensemble average of the parameter N µ is calculated by
using equation 2.20. Subsequently, the expectation value of the suf�cient
statistics hr(x, z)i N µ , 2.19, is calculated by using the posterior distribution for
the hidden variable P(z(t)|x(t), N µ ), equation 2.14. In the VB M-step, the posterior
hyperparameter ® is updated by using equation 2.18. Repeating this
process, the free energy function, equation 2.26, increases monotonically.
This process continues until the free energy function converges.
Using equations 2.28 and 2.30, VB equations 2.20 and 2.18 can be expressed
as the gradient method:
D N µ D N µnew ¡ N µ D hµi ® ¡ N µ
D U ¡1 ( µ
N ) @F
@N (XfTg,
µ
N µ, ®, c ), (2.32)