Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.526 43 — Boltzmann MachinesThe first term 〈y i y j 〉 P (h | x (n) ,W) is the correlation between y i and y j if theBoltzmann machine is simulated with the visible variables clamped to x (n)and the hidden variables freely sampling from their conditional distribution.The second term 〈y i y j 〉 P (x,h | W)is the correlation between y i and y j whenthe Boltzmann machine generates samples from its model distribution.Hinton and Sejnowski demonstrated that non-trivial ensembles such asthe labelled shifter ensemble can be learned using a Boltzmann machine withhidden units. The hidden units take on the role of feature detectors that spotpatterns likely to be associated with one of the three shifts.The Boltzmann machine is time-consuming to simulate because the computationof the gradient of the log likelihood depends on taking the difference oftwo gradients, both found by Monte Carlo methods. So Boltzmann machinesare not in widespread use. It is an area of active research to create modelsthat embody the same capabilities using more efficient computations (Hintonet al., 1995; Dayan et al., 1995; Hinton and Ghahramani, 1997; Hinton, 2001;Hinton and Teh, 2001).43.3 Exercise⊲ Exercise 43.3. [3 ] Can the ‘bars and stripes’ ensemble (figure 43.2) be learnedby a Boltzmann machine with no hidden units? [You may be surprised!]Figure 43.2. Four samples fromthe ‘bars and stripes’ ensemble.Each sample is generated by firstpicking an orientation, horizontalor vertical; then, for each row ofspins in that orientation (each baror stripe respectively), switchingall spins on with probability 1/ 2.