Foundations of Data Science

x 1
x 1 + x 2 + x 3 x 3 + x 4 + x 5 x 4 x 5
x 1 x 2 x 3 x 4 x 5
Figure 9.3: The factor graph for the function f = x 1 (x 1 + x 2 + x 3 ) (x 3 + x 4 + x 5 ) x 4 x 5 .
Consider the following example where
f = x 1 (x 1 + x 2 + x 3 ) (x 3 + x 4 + x 5 ) x 4 x 5
and the variables take on values 0 or 1. Consider marginalizing f by computing
f (x 1 ) =
∑
x 1 (x 1 + x 2 + x 3 ) (x 3 + x 4 + x 5 ) x 4 x 5 ,
x 2 x 3 x 4 x 5
In this case the factor graph is a tree as shown in Figure 9.3. The factor graph as a
rooted tree and the messages passed by each node to its parent are shown in Figure 9.4.
If instead of computing marginal’s, one wanted the variable assignment that maximizes
the function f, one would modify the above procedure by replacing the summation by a
maximization operation. Obvious modifications handle the situation where f(x) is a sum
of products.
f (x) =
∑
g (x)
x 1 ,...,x n
9.8 Message Passing in general Graphs
The simple message passing algorithm in the last section gives us the one variable
function of x 1 when we sum out all the other variables. For a general graph that is not
a tree, we formulate an extension of that algorithm. But unlike the case of trees, there
is no proof that the algorithm will converge and even if it does, there is no guarantee
that the limit is the marginal probability. This has not prevented its usefulness in some
applications.
First, lets ask a more general question, just for trees. Suppose we want to compute
for each i the one variable function of x i when we sum out all variables x j , j ≠ i. Do we
have to repeat what we did for x 1 once for each x i ? Luckily, the answer is no. It will
suffice to do a second pass from the root to the leaves of essentially the same message
passing algorithm to get all the answers. Recall that in the first pass, each edge of the
tree has sent a message “up”, from the child to the parent. In the second pass, each edge
will send a message down from the parent to the child. We start with the root and work
313