Foundations of Data Science

computes the marginal probabilities and thus the word marginalization. The second problem
involves computing the assignment to the variables that maximizes the function f.
When f is a probability distribution, this problem is the maximum a posteriori probability
or MAP problem.
If the factor graph is a tree, then there exists an efficient algorithm for solving these
problems. Note that there are four problems: the function f is either a product or a sum
and we are either marginalizing or finding the maximizing assignment to the variables. All
four problems are solved by essentially the same algorithm and we present the algorithm
for the marginalization problem when f is a product. Assume we want to “sum out” all
the variables except x 1 , leaving a function of x 1 .
Call the variable node associated with the variable x i node x i . First, make the node x 1
the root of the tree. It will be useful to think of the algorithm first as a recursive algorithm
and then unravel the recursion. We want to compute the product of all factors occurring
in the sub-tree rooted at the root with all variables except the root-variable summed out.
Let g i be the product of all factors occurring in the sub-tree rooted at node x i with all
variables occurring in the subtree except x i summed out. Since this is a tree, x 1 will not
reoccur anywhere except the root. Now, the grandchildren of the root are variable nodes
and suppose for recursion, each grandchild x i of the root, has already computed its g i . It
is easy to see that we can compute g 1 as follows.
Each grandchild x i of the root passes its g i to its parent, which is a factor node. Each
child of x 1 collects all its children’s g i , multiplies them together with its own factor and
sends the product to the root. The root multiplies all the products it gets from its children
and sums out all variables except its own variable, namely here x 1 .
Unraveling the recursion is also simple, with the convention that a leaf node just receives
1, product of an empty set of factors, from its children. Each node waits until it
receives a message from each of its children. After that, if the node is a variable node,
it computes the product of all incoming messages, and sums this product function over
all assignments to the variables except for the variable of the node. Then, it sends the
resulting function of one variable out along the edge to its parent. If the node is a factor
node, it computes the product of its factor function along with incoming messages from
all the children and sends the resulting function out along the edge to its parent.
The reader should prove that the following invariant holds assuming the graph is a tree:
Invariant The message passed by each variable node to its parent is the product of
all factors in the subtree under the node with all variables in the subtree except its own
summed out.
312