Bernal S D_2010.pdf - University of Plymouth

3.4. EXISTING MODELS
maintaining the 2-layer structure characteristic of Restricted Boltzmann Machines.
The above learning method can be seen as a variational approximation wherein the constraint
is thai the weights in the higher levels ensure the complementary priors condition, therefore
yielding a factorial posterior distribution. However, as weights in higher-levels are learned, the
priors for lower layers cease to be complementary, so ihe weights used during inference are
incorrect. Nonetheless, it can be shown that each time the weights of a layer are adapted, the
variational lower bound on ihe log probability of the training data is improved, consequently
improving the overall generative model. The weights of ihe model are then linely tuned in a
final stage by performing an up and down pass of a variant of the wake-sleep algorithm (Hinton
et al. 1995). Although Ihe learning is unsupervised in the directed layers, the lop two associative
layers can be used to learn labeled data.
Inference is achieved by a single up pass along the bottom directed layers, yielding the binary
states of Ihe units in the lower associative memory layer. Further Gibbs sampling or free-
energy optimization activates the correct label unit at the top layer. The pertbmiance of the
model on the MNIST digit recognition task was superior to that of previous models, including
Support Vector Machines and back-propagation. This demonstrates that generative models can
learn many more parameters than discriminative models without overfilting. The model is still
limited in that top-down feedback during inference is restricted to the top associative layers.
Additionally, it does noi deal systematically with perceptual invariances. Instead, invariance
arises as a consequence of the wide range of sample images that can be generated by the model
for each given category,
3.4.2.3 Models based on variatiimul approximation methods
The free-energy model proposed by rri>ilon (Frislon 200;i. 2('H),'i, Friston et al. 2(M)6, Frislon and
Stephan 2007. Friston and Kiebel 2()()9, Frislon 2010) has already been described in some detail
in Section 3,l.,l. It is based on a variational approximation and therefore converts the complex
inference problem into an optimization task which tries to minimize the free-energy between
the true posterior distribution and and the recognition distribution. By assuming a Gaussian ap­
proximation (l-aplace assumption) to the recognition distribution, optimization becomes equiv-
129