Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.514 42 — Hopfield Networks10.80.60.40.200 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.1610.990.980.970.960.950.09 0.1 0.11 0.12 0.13 0.14 0.15Figure 42.8. Overlap between adesired memory and the stablestate nearest to it as a function ofthe loading fraction N/I. Theoverlap is defined to be the scaledinner product ∑ i x ix (n)i /I, whichis 1 when recall is perfect and zerowhen the stable state has 50% ofthe bits flipped. There is anabrupt transition at N/I = 0.138,where the overlap drops from 0.97to zero.whereΦ(z) ≡∫ z−∞dz1√2πe −z2 /2 . (42.23)The important quantity N/I is the ratio of the number of patterns stored tothe number of neurons. If, for example, we try to store N ≃ 0.18I patternsin the Hopfield network then there is a chance of 1% that a specified bit in aspecified pattern will be unstable on the first iteration.We are now in a position to derive our first capacity result, for the casewhere no corruption of the desired memories is permitted.⊲ Exercise 42.7. [2 ] Assume that we wish all the desired patterns to be completelystable – we don’t want any of the bits to flip when the network is putinto any desired pattern state – and the total probability of any error atall is required to be less than a small number ɛ. Using the approximationto the error function for large z,Φ(−z) ≃ √ 1 e −z2 /2, (42.24)2π zshow that the maximum number of patterns that can be stored, N max ,isIN max ≃4 ln I + 2 ln(1/ɛ) . (42.25)If, however, we allow a small amount of corruption of memories to occur, thenumber of patterns that can be stored increases.The statistical physicists’ capacityThe analysis that led to equation (42.22) tells us that if we try to store N ≃0.18I patterns in the Hopfield network then, starting from a desired memory,about 1% of the bits will be unstable on the first iteration. Our analysis doesnot shed light on what is expected to happen on subsequent iterations. Theflipping of these bits might make some of the other bits unstable too, causingan increasing number of bits to be flipped. This process might lead to anavalanche in which the network’s state ends up a long way from the desiredmemory.In fact, when N/I is large, such avalanches do happen. When N/I is small,they tend not to – there is a stable state near to each desired memory. For thelimit of large I, Amit et al. (1985) have used methods from statistical physicsto find numerically the transition between these two behaviours. There is asharp discontinuity atN crit = 0.138I. (42.26)