Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.26.1: The general problem 335The function P (x), by the way, may be recognized as the posterior probabilitydistribution of the three transmitted bits in a repetition code (section1.2) when the received signal is r = (1, 1, 0) and the channel is a binary symmetricchannel with flip probability 0.1. The factors f 4 and f 5 respectivelyenforce the constraints that x 1 and x 2 must be identical and that x 2 and x 3must be identical. The factors f 1 , f 2 , f 3 are the likelihood functions contributedby each component of r.A function of the factored form (26.1) can be depicted by a factor graph, inwhich the variables are depicted by circular nodes and the factors are depictedby square nodes. An edge is put between variable node n and factor node mif the function f m (x m ) has any dependence on variable x n . The factor graphfor the example function (26.4) is shown in figure 26.1.x 1 x 2 x 3❣ ❣ ❣✥✥✥✥✥✥✥✥✥✥✥✥✥✥✥✥✥✥✥✥✥❅ ❅ f 1 f 2 f 3 f 4 f 5Figure 26.1. The factor graphassociated with the functionP ∗ (x) (26.4).The normalization problemThe first task to be solved is to compute the normalizing constant Z.The marginalization problemsThe second task to be solved is to compute the marginal function of anyvariable x n , defined by∑Z n (x n ) = P ∗ (x). (26.5){x n ′ }, n ′ ≠nFor example, if f is a function of three variables then the marginal forn = 1 is defined byZ 1 (x 1 ) = ∑ x 2,x 3f(x 1 , x 2 , x 3 ). (26.6)This type of summation, over ‘all the x n ′ except for x n ’ is so important that itcan be useful to have a special notation for it – the ‘not-sum’ or ‘summary’.The third task to be solved is to compute the normalized marginal of anyvariable x n , defined by∑P n (x n ) ≡ P (x). (26.7){x n ′ }, n ′ ≠n[We include the suffix ‘n’ in P n (x n ), departing from our normal practice in therest of the book, where we would omit it.]⊲ Exercise 26.1. [1 ] Show that the normalized marginal is related to the marginalZ n (x n ) byP n (x n ) = Z n(x n )Z . (26.8)We might also be interested in marginals over a subset of the variables,such asZ 12 (x 1 , x 2 ) ≡ ∑ x 3P ∗ (x 1 , x 2 , x 3 ). (26.9)All these tasks are intractable in general. Even if every factor is a functionof only three variables, the cost of computing exact solutions for Z and forthe marginals is believed in general to grow exponentially with the number ofvariables N.For certain functions P ∗ , however, the marginals can be computed efficientlyby exploiting the factorization of P ∗ . The idea of how this efficiency