Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.42Hopfield NetworksWe have now spent three chapters studying the single neuron. The time hascome to connect multiple neurons together, making the output of one neuronbe the input to another, so as to make neural networks.Neural networks can be divided into two classes on the basis of their connectivity.Figure 42.1. (a) A feedforwardnetwork. (b) A feedback network.(a)(b)Feedforward networks. In a feedforward network, all the connections aredirected such that the network forms a directed acyclic graph.Feedback networks. Any network that is not a feedforward network will becalled a feedback network.In this chapter we will discuss a fully connected feedback network calledthe Hopfield network. The weights in the Hopfield network are constrained tobe symmetric, i.e., the weight from neuron i to neuron j is equal to the weightfrom neuron j to neuron i.Hopfield networks have two applications. First, they can act as associativememories. Second, they can be used to solve optimization problems. We willfirst discuss the idea of associative memory, also known as content-addressablememory.42.1 Hebbian learningIn Chapter 38, we discussed the contrast between traditional digital memoriesand biological memories. Perhaps the most striking difference is the associativenature of biological memory.A simple model due to Donald Hebb (1949) captures the idea of associativememory. Imagine that the weights between neurons whose activities arepositively correlated are increased:dw ijdt∼ Correlation(x i , x j ). (42.1)Now imagine that when stimulus m is present (for example, the smell of abanana), the activity of neuron m increases; and that neuron n is associated505