Online Model Selection Based on the Variational Bayes

<strong>Online</strong> <strong>Model</strong> <strong>Selection</strong> <strong>Based</strong> on the Variational Bayes 1667
This method is used in a model selection task for dynamic environments in
section 5.
In the next section, we study the model selection problem for mixture
of gaussian models. As a mechanism for structural change, we adopt the
split-and-merge method proposed by Ueda et al. (1999) (see also Richardson
& Green, 1997; Ghahramani & Beal, 2000; Ueda, 1999). For mixture models,
the split-and-merge method provides a simple procedure for structural
changes. We choose either to split a unit into two or to merge two units
into one in the sequential model selection process. In the current implementation,
the same process is applied if the previous attempt was successful.
Otherwise the other process is applied.
A criterion for splitting a unit is given by the unit’s free energy, which is
assigned to each unit (see appendix C). The split is applied to the unit with
the lowest free energy among unattempted units. A criterion for merging
units is given by the correlation between the two units’ activities, which are
represented by the posterior probability that the units will be selected for
given data. The unit pair with the highest correlation among unattempted
unit pairs is selected for merging. The deletion of units is also performed
for units with very small activities, which indicate that the units have not
been selected at all.
We adopted the above model selection procedure because of its simplicity.
Other model selection procedures using the split-and-merge algorithm
have also been proposed (Ghahramani & Beal, 2000; Ueda, 1999).
By combining the sequential model selection procedure with the online
VB learning method, a fully online learning method with a model selection
mechanism is obtained, and it can be applied to real-time applications.
5 Experiments
As a preliminary study on the performance of the online VB method, we considered
model selection problems for two-dimensional mixture of gaussian
(MG) models (see appendix C). We borrowed two tasks from Roberts et al.
(1998). Data set A, consisting of 200 points, was generated from a mixture
of four gaussians with the centers (0, 0), (2, p 12), (4, 0), and (¡2, ¡ p 12)
(see Figure 1A). The gaussians had the same isotropic variance s 2 D (1.2) 2 .
In addition, data set B, consisting of 1000 points, was generated from a
mixture of four gaussians (see Figure 1B). In this case, they were paired
such that each pair had a common center—m1 D m2 D (2, p 12) and
m3 D m4 D (¡2, ¡ p 12)—but different variances—s 2 1 D s2 3 D (1.0)2 and
s 2 2 D s2 4 D (5.0)2 . Although these models were simple, the model selection
tasks for them were rather dif�cult because of the overlap between the
gaussians (Roberts et al., 1998).
In the �rst experiment, we examined the usual Bayes model selection
procedure. A set of models consisting of different numbers of units was