Bernal S D_2010.pdf - University of Plymouth

3.3. D^JmnON AND MATHEMATICAL F0RMU1.ATI0N
pass for each branch, such that for every connecting arc in the network only one X and one n
messages are required to effectively obtain the exact beliefs (e.g. Figure 3.6).
Note Uiat for a singly-connected network lo have nodes with multiple parents, these parents
niiisi not be interconnected, otherwi.se it becomes the third type of network, i.e. a multiply-
connected network. In these networks the number of steps required is not fixed as mes.sages
circulate indelinitely. Thus, messages from intermediate nodes are typically initialized to a
Mat distribution and propagate upwards and downwards simultaneously from the first time-step
onwards. Feedback in multiply connected networks is described in more detail in .Section 3.3.5.
In singly-connected networks, although messages from the root and intermediate nodes could be
initialized to a fiat distribution and propagated from the first time step, these would just generate
temporal beliefs that would not affect the final exact belief. To avoid these extra calculations,
belief propagation dictates that nodes only generate output messages once they have received
all the required incoming messages. This means the 7t messages (red arrows) in steps I and
2 of Figure 3.9 are redundant, i.e. they don't contribute to the final belief. For this reason,
in singly-connected networks, belief propagation can be argued to occur in a single bottom-up
and a top-down pass. Another important property of Ibis type of network is that the message
propagation scheme can be implemented asynchronously, in other words, it does not require
any particular order to provide the correct beliefs.
3.3.4 Combining messages from multiple parents
In discrete Bayesian networks, the conditional probability table (CPT) which relates states of the
parent nodes to those of a child node includes entries for all possible combinations of the child
and parent node stales. Given a node X with kx stales, and its parent ntxJes lJ\,...Mtj with ku
states each, the number of entries in the CPT is equal to kx -k^. This means the number of entries
is exponential lo the number of parents, such that even for relatively moderate dimensions
(kx —ku-4,N = 8), the size of the CPT becomes large and unmanageable (262,144 entries).
Not only does the storage space increase exponentially with the number of parents, but so does
the compulation time required to compute the belief and the messages at node X. Additionally,
learning all the values of the CPT can be problematic as the training data may not include all
106