Bernal S D_2010.pdf - University of Plymouth

3J. DEFINITION AND MATHEMATICAL FORMULATION
large-scale networks (Jordan and Weiss 2002).
• Cutset conditioning^: This method, also cjilled reasoning by assumplion. also provides
the exact marginal probabilities. It involves breaking the loops by finding a small set of
variables which, if known (i.e. instantiated), would render the remaining graph singly
connected. For each value of these variables, belief propagation obtains the beliefs of the
nodes in the the singly connected network. The final value is obtained by averaging the
resulting beliefs with the appropriate weights obtained from the normalization constants.
3.3.5.2 Appruximale inl'erencc methods
• hiopy belief propagation: This method implies naively applying the belief propagation
algorithm on a network despite it having loops. The formulation would be theoretically
incorrect, and the messages would circulate indefinitely through the network due to its
recursive nature. Nonetheless, empirical results in error-correciing networks, such as
the turbo code (Weiss 1997), demonstrate the method provides a good approximation
10 the correct beUefs. The method has also been applied satisfactorily to other type of
network structures, such as the P VRAM ID network, which resembles those used for i mage
processing (Murphy ct al. 1999, Weiss 2000). The resulting beliefs in these networks
showed convergence, as opposed lo oscillations, after a number of iterations,
• SampHnfi/Monle-Carlo algorithms: These methods rely on the fact that, while it might be
infeasible lo compute the exact belief distribution, it may be possible to obtain samples
from it, or from a closely-related distribution, such that the belief can be approximated
averaging over ihesc samples. For large deep networks these methods can be very slow
(Hintonelal.2006).
The Gihhs sampling and Metropolis-Hastings algorithm are both special cases of the
Markov Chain Monte Carlo algorithm. The first one involves selecting a variable, x\
for example, and computing a simplified version of its belief based only on the slate of
its neighbours at time f, such that Be/(y,"^') = P{J:||A2, ..•,.*1,)' The priKcss is repealed
for all variables using always the latest (most recently updated) value for its neighboura.
112