Bernal S D_2010.pdf - University of Plymouth

3.4. EXISTING MODELS
by the higher levels to infer additional characteristics of the image, such as the precise identity
of the face.
Lewicki and Sejnowski (1997) demonstrate the efficiency of Gibbs sampling in learning higher
level parameters in Bayesian networks. A simple 3-ievcl network of sl(x;hasiic binary variables
with a 5x5 pixel input image is used lo discover higher level motion patterns from the inpiil
image correlations (the Shifter problem). Importantly, feedback from the third layer, containing
the global direction of motion, is used lo disambiguate the local shift direction in layer two. The
combination of information from multiple parents/causes was approximated using the Noisy-
C)R gate, previously described in Section 3.3.4.
I linton et al. (2006) proposed a new type of network called a deep belief nei which is composed
of a Bayesian network (directed acyclic graph) with two undirected associative memory layers
al the lop. The motivation for this model is lo ease the inlraciable unsupervised leaminj; pro­
cess in hierarchical Bayesian networks, where, in order lo learn the weights of the bonom layer
il is necessary to calculate the posterior probability which depends not only on the likelihood
(bottom-up data) but also on the prior (top-down data). In other words, as a result of the ex­
plaining away effect, the weights of all the higher layers are required. Turther. it is necessary to
sum over all possible configurations of the higher variables in order to obtain the bottom layer
prior.
The authors introduce the concept o( complementary priurs, which are prior distributions that,
when multiplied by the corresponding likelihood function, yield a posterior distribution which
can be factorized. This implies eliminating the expiaining-away effect, thus making each hid­
den layer independent of its parents' weights. This yields a network which is equivalent to
a Restricted Boll/mann Machine, i.e. a network with an independent hidden layer of binary
variables with undirected symmetric connections to a layer of observed nodes. Under these
conditions a fast learning algorithm is derived which obtains the approximate parameters of the
network layer by layer. First, a visible layer (input image) is used to train the bottom hidden
layer of the network. After learning the weights of the hidden layer, the activations of that
layer, given the input image, are used as the input data for the hidden layer above, thus always
128