Foundations of Data Science

x 1 + x 2 + x 3 x 1 + x 2 x 1 + x 3 x 2 + x 3
x 1 x 2 x 3
Figure 9.2: The factor graph for the function
f(x 1 , x 2 , x 3 ) = (x 1 + x 2 + x 3 )(x 1 + ¯x 2 )(x 1 + ¯x 3 )(¯x 2 + ¯x 3 ).
In general, the f i are not convex; indeed they may be discrete. So the minimization
cannot be carried out by a known polynomial time algorithm. The most used forms of the
Markov random field involve S i which are cliques of a graph. So we make the following
definition.
A Markov Random Field consists of an undirected graph and an associated function
that factorizes into functions associated with the cliques of the graph. The special case
when all the factors correspond to cliques of size one or two is of interest.
9.6 Factor Graphs
Factor graphs arise when we have a function f of a variables x = (x 1 , x 2 , . . . , x n ) that
can be expressed as f (x) = ∏ f α (x α ) where each factor depends only on some small
α
number of variables x α . The difference from Markov random fields is that the variables
corresponding to factors do not necessarily form a clique. Associate a bipartite graph
where one set of vertices correspond to the factors and the other set to the variables.
Place an edge between a variable and a factor if the factor contains that variable. See
Figure 9.2
9.7 Tree Algorithms
Let f(x) be a function that is a product of factors. When the factor graph is a tree
there are efficient algorithms for solving certain problems. With slight modifications, the
algorithms presented can also solve problems where the function is the sum of terms rather
than a product of factors.
The first problem is called marginalization and involves evaluating the sum of f over
all variables except one. In the case where f is a probability distribution the algorithm
311