Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.516 42 — Hopfield Networksw = x’ * x ;for l = 1:L# initialize the weights using Hebb rule# loop L timesfor i=1:I #w(i,i) = 0 ; # ensure the self-weights are zero.end #a = x * w ; # compute all activationsy = sigmoid(a) ; # compute all outputse = t - y ; # compute all errorsgw = x’ * e ; # compute the gradientsgw = gw + gw’ ; # symmetrize gradientsAlgorithm 42.9. Octave sourcecode for optimizing the weights ofa Hopfield network, so that itworks as an associative memory.cf. algorithm 39.5. The datamatrix x has I columns and Nrows. The matrix t is identical tox except that −1s are replaced by0s.w = w + eta * ( gw - alpha * w ) ; # make stependforSo, just as we defined an objective function (39.11) for the training of asingle neuron as a classifier, we can definewhereandG(W) = − ∑ iy (n)i=∑nt (n)iln y (n)it (n)i=11 + exp(−a (n)i)+ (1 − t (n)i) ln(1 − y (n)i) (42.28){1 x (n)i= 10 x (n)i= −1, where a(n)i(42.29)= ∑ w ij x (n)j. (42.30)We can then steal the algorithm (algorithm 39.5, p.478) which we wrote forthe single neuron, to write an algorithm for optimizing a Hopfield network,algorithm 42.9. The convenient syntax of Octave requires very few changes;the extra lines enforce the constraints that the self-weights w ii should all bezero and that the weight matrix should be symmetrical (w ij = w ji ).As expected, this learning algorithm does a better job than the one-shotHebbian learning rule. When the six patterns of figure 42.5, which cannot bememorized by the Hebb rule, are learned using algorithm 42.9, all six patternsbecome stable states.Exercise 42.8. [4C ] Implement this learning rule and investigate empirically itscapacity for memorizing random patterns; also compare its avalancheproperties with those of the Hebb rule.42.9 Hopfield networks for optimization problemsSince a Hopfield network’s dynamics minimize an energy function, it is naturalto ask whether we can map interesting optimization problems onto Hopfieldnetworks. Biological data processing problems often involve an element ofconstraint satisfaction – in scene interpretation, for example, one might wishto infer the spatial location, orientation, brightness and texture of each visibleelement, and which visible elements are connected together in objects. Theseinferences are constrained by the given data and by prior knowledge aboutcontinuity of objects.