Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.26Exact Marginalization in GraphsWe now take a more general view of the tasks of inference and marginalization.Before reading this chapter, you should read about message passing in Chapter16.26.1 The general problemAssume that a function P ∗ of a set of N variables x ≡ {x n } N n=1a product of M factors as follows:is defined asP ∗ (x) =M∏f m (x m ). (26.1)m=1Each of the factors f m (x m ) is a function of a subset x m of the variables thatmake up x. If P ∗ is a positive function then we may be interested in a secondnormalized function,P (x) ≡ 1 Z P ∗ (x) = 1 ZM∏f m (x m ), (26.2)m=1where the normalizing constant Z is defined byZ = ∑ xM∏f m (x m ). (26.3)m=1As an example of the notation we’ve just introduced, here’s a function ofthree binary variables x 1 , x 2 , x 3 defined by the five factors:f 1 (x 1 ) =f 2 (x 2 ) =f 3 (x 3 ) =f 4 (x 1 , x 2 ) =f 5 (x 2 , x 3 ) ={ 0.1 x1 = 0{0.9 x 1 = 10.1 x2 = 0{0.9 x 2 = 10.9 x3 = 0{0.1 x 3 = 11 (x1 , x 2 ) = (0, 0) or (1, 1){0 (x 1 , x 2 ) = (1, 0) or (0, 1)1 (x2 , x 3 ) = (0, 0) or (1, 1)0 (x 2 , x 3 ) = (1, 0) or (0, 1)P ∗ (x) = f 1 (x 1 )f 2 (x 2 )f 3 (x 3 )f 4 (x 1 , x 2 )f 5 (x 2 , x 3 )P (x) = 1 Z f 1(x 1 )f 2 (x 2 )f 3 (x 3 )f 4 (x 1 , x 2 )f 5 (x 2 , x 3 ).(26.4)The five subsets of {x 1 , x 2 , x 3 } denoted by x m in the general function (26.1)are here x 1 = {x 1 }, x 2 = {x 2 }, x 3 = {x 3 }, x 4 = {x 1 , x 2 }, and x 5 = {x 2 , x 3 }.334