Online Model Selection Based on the Variational Bayes

<strong>Online</strong> <strong>Model</strong> <strong>Selection</strong> <strong>Based</strong> on the Variational Bayes 1659
By interchanging the integration with respect to µ and z, one can get
Z
P(x|XfTg) D dm (z)
£ exp £ r0(x, z) C © ( O® (x, z), c C 1) ¡ © (®, c ) ¤ , (2.37)
O® (x, z) D (c ® C r(x, z))/(1 C c ).
For a �nite T, this predictive distribution has a different functional form
from the model distribution P(x| µ ), equation 2.1.
2.7 Large Sample Limit. When the amount of observed data becomes
large (T À 1 : c À 1), the solution of the VB algorithm becomes the ML
estimator (Attias, 1999). In this limit, the integration over the parameters
with respect to the posterior parameter distribution can be approximated
by using a stationary point approximation:
Z
exp [© (®, c )] D
dm (µ ) exp [c (a ¢ µ ¡ Ã (µ ))]
µ
� exp c (a ¢ O µ ¡ Ã ( µ
O ) ¡ 1
2 log
�
�
�
�c @2Ã @µ@µ ( �
µ
O
�
) �
� C O(1/c
) , (2.38)
where O µ is the maximum of the exponent ¡ ® ¢ µ ¡ Ã (µ )¢ , that is,
@Ã
@µ ( O µ ) D ®. (2.39)
Therefore, © can be approximated as
© (®, c ) � c (a ¢ O µ ¡ Ã ( µ
O ) ¡ 1
2 log
�
�
�
�c @2Ã @µ@µ ( �
µ
O
�
) �
� C O(1/c ). (2.40)
Consequently, the ensemble average of the parameter N µ can be approximated
as
Nµ D 1
c
� 1
c
@©
@® (®, c )
@
@® (c (® ¢ O µ ¡ Ã ( O µ ))) D O µ. (2.41)
The relations 2.39 and 2.41 imply that the posterior hyperparameter ® is
equal to the expectation parameter of the EFH model, Á (see equation 2.3)
in this limit. Furthermore, equations 2.18, 2.39, and 2.41 are equivalent to the