Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.246 16 — Message Passingpath to a node is achieved by more than one route to it, then the algorithmcan pick any of those routes at random.]We can recover this lowest-cost path by backtracking from B, followingthe trail of surviving edges back to A. We deduce that the lowest-cost path isA–I–K–M–B.Other applications of the min–sum algorithmImagine that you manage the production of a product from raw materialsvia a large set of operations. You wish to identify the critical path in yourprocess, that is, the subset of operations that are holding up production. Ifany operations on the critical path were carried out a little faster then thetime to get from raw materials to product would be reduced.The critical path of a set of operations can be found using the min–sumalgorithm.In Chapter 25 the min–sum algorithm will be used in the decoding oferror-correcting codes.16.4 Summary and related ideasSome global functions have a separability property. For example, the numberof paths from A to P separates into the sum of the number of paths from A to M(the point to P’s left) and the number of paths from A to N (the point aboveP). Such functions can be computed efficiently by message-passing. Otherfunctions do not have such separability properties, for example1. the number of pairs of soldiers in a troop who share the same birthday;2. the size of the largest group of soldiers who share a common height(rounded to the nearest centimetre);3. the length of the shortest tour that a travelling salesman could take thatvisits every soldier in a troop.One of the challenges of machine learning is to find low-cost solutions to problemslike these. The problem of finding a large subset of variables that areapproximately equal can be solved with a neural network approach (Hopfieldand Brody, 2000; Hopfield and Brody, 2001). A neural approach to the travellingsalesman problem will be discussed in section 42.9.16.5 Further exercises⊲ Exercise 16.3. [2 ] Describe the asymptotic properties of the probabilities depictedin figure 16.11a, for a grid in a triangle of width and height N.(a)2 J 24 ✟ ✟✯ ❍ ❍❥4 1 MH 2 10 ✟ ✟✯ ❍ ❍❥KBA ❍✟ ✟✯ ❍ ❍❥21 ❍❥1 ✟ ✟✯ ❍1 ❍❥N 3I ❍✟ ✟✯1 ❍❥L ✟ ✟✯3(b)62 2J4 ✟ ✟✯ ❍ ❍❥4 1 MH 2 10 ✟ ✟✯ ❍ ❍❥ 5K ✟ ✟✯ ❍ ❍❥BA ❍ 2 31 ❍❥1 ✟ ✟✯ ❍1 ❍❥N ✟ ✟✯3I ❍1 ❍❥2 ✟ ✟✯3(c)62 2J4 ✟ ✟✯ ❍ ❍❥4 1 MH 2 10 ✟ ✟✯ 3BA ❍✟ ✟✯ ❍ ❍❥2 K1 ❍❥1 ✟ ✟✯ ❍1 ❍❥N ✟ ✟✯3I ❍1 ❍❥2 ✟ ✟✯3(d)2 24 ✟ ✟✯ 54 1H 2 1M0 ✟ ✟✯ 3BA ❍✟ ✟✯ ❍ ❍❥2 K1 ❍❥1 ✟ ✟✯ ❍1 ❍❥ ✟✯4 3I ❍ ✟N1 ❍❥3LL6J2L6J(e)2 24 ✟ ✟✯ 54 1H 2 1M0 ✟ ✟✯ 3 ✟ ✟✯ ❍ ❍❥A ❍ 2 K1 ❍❥1 ✟ ✟✯ ❍1 ❍❥4 3I ❍ N1 ❍❥32L6BFigure 16.13. Min–summessage-passing algorithm to findthe cost of getting to each node,and thence the lowest cost routefrom A to B.⊲ Exercise 16.4. [2 ] In image processing, the integral image I(x, y) obtained froman image f(x, y) (where x and y are pixel coordinates) is defined byx∑ y∑I(x, y) ≡ f(u, v). (16.1)u=0 v=0Show that the integral image I(x, y) can be efficiently computed by messagey 2passing.Show that, from the integral image, some simple functions of the imagey 1x 1 x 2can be obtained. For example, give an expression for the sum of theimage intensities f(x, y) for all (x, y) in a rectangular region extending(0, 0)from (x 1 , y 1 ) to (x 2 , y 2 ).