Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.512 42 — Hopfield NetworksDesired memoriesmoscow------russialima----------perulondon-----englandtokyo--------japanedinburgh-scotlandottawa------canadaoslo--------norwaystockholm---swedenparis-------france→ W →Attracting stable statesmoscow------russialima----------perulondog-----englard (1)tonco--------japan (1)edinburgh-scotland(2)oslo--------norwaystockholm---swedenparis-------francewzkmhewn--xqwqwpoq (3)paris-------sweden (4)ecnarf-------sirap (4)where f(a) is the activation function, for example f(a) = tanh(a). For asteady activation a i , the activity x i (t) relaxes exponentially to f(a i ) withtime-constant τ.Now, here is the nice result: as long as the weight matrix is symmetric,this system has the variational free energy (42.15) as its Lyapunov function.Figure 42.6. Failure modes of aHopfield network (highlyschematic). A list of desiredmemories, and the resulting list ofattracting stable states. Notice(1) some memories that areretained with a small number oferrors; (2) desired memories thatare completely lost (there is noattracting stable state at thedesired memory or near it); (3)spurious stable states unrelated tothe original list; (4) spuriousstable states that areconfabulations of desiredmemories.⊲ Exercise 42.6. [1 ] By computing d dt ˜F , prove that the variational free energy˜F (x) is a Lyapunov function for the continuous-time Hopfield network.It is particularly easy to prove that a function L is a Lyapunov function if thesystem’s dynamics perform steepest descent on L, with d dt x i(t) ∝∂∂x iL. Inthe case of the continuous-time continuous Hopfield network, it is not quitedso simple, but every component ofdt x ∂i(t) does have the same sign as∂x i˜F ,which means that with an appropriately defined metric, the Hopfield networkdynamics do perform steepest descents on ˜F (x).42.7 The capacity of the Hopfield networkOne way in which we viewed learning in the single neuron was as communication– communication of the labels of the training data set from one point intime to a later point in time. We found that the capacity of a linear thresholdneuron was 2 bits per weight.Similarly, we might view the Hopfield associative memory as a communicationchannel (figure 42.6). A list of desired memories is encoded into aset of weights W using the Hebb rule of equation (42.5), or perhaps someother learning rule. The receiver, receiving the weights W only, finds thestable states of the Hopfield network, which he interprets as the original memories.This communication system can fail in various ways, as illustrated inthe figure.1. Individual bits in some memories might be corrupted, that is, a stablestate of the Hopfield network is displaced a little from the desiredmemory.2. Entire memories might be absent from the list of attractors of the network;or a stable state might be present but have such a small basin ofattraction that it is of no use for pattern completion and error correction.3. Spurious additional memories unrelated to the desired memories mightbe present.4. Spurious additional memories derived from the desired memories by operationssuch as mixing and inversion may also be present.