Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.43Boltzmann Machines43.1 From Hopfield networks to Boltzmann machinesWe have noticed that the binary Hopfield network minimizes an energy functionE(x) = − 1 2 xT Wx (43.1)and that the continuous Hopfield network with activation function x n =tanh(a n ) can be viewed as approximating the probability distribution associatedwith that energy function,P (x | W) = 1Z(W) exp[−E(x)] = 1 [ ] 1Z(W) exp 2 xT Wx . (43.2)These observations motivate the idea of working with a neural network modelthat actually implements the above probability distribution.The stochastic Hopfield network or Boltzmann machine (Hinton and Sejnowski,1986) has the following activity rule:Activity rule of Boltzmann machine: after computing the activationa i (42.3),set x i = +1 with probabilityelse set x i = −1.11 + e −2a i(43.3)This rule implements Gibbs sampling for the probability distribution (43.2).Boltzmann machine learningGiven a set of examples {x (n) } N 1 from the real world, we might be interestedin adjusting the weights W such that the generative modelP (x | W) = 1 [ ] 1Z(W) exp 2 xT Wx(43.4)is well matched to those examples. We can derive a learning algorithm bywriting down Bayes’ theorem to obtain the posterior probability of the weightsgiven the data:[ N]∏P (x (n) | W) P (W)P (W | {x (n) } N 1 }) = n=1P ({x (n) } N 1 }) . (43.5)522