Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.518 42 — Hopfield NetworksFigure 42.11. (a) Evolution of thestate of a continuous Hopfieldnetwork solving a travellingsalesman problem using Aiyer’s(1991) graduated non-convexitymethod; the state of the networkis projected into thetwo-dimensional space in whichthe cities are located by findingthe centre of mass for each pointin the tour, using the neuronactivities as the mass function.(b) The travelling scholarproblem. The shortest tourlinking the 27 CambridgeColleges, the EngineeringDepartment, the UniversityLibrary, and Sree Aiyer’s house.From Aiyer (1991).(a)(b)tour, plus a constant given by the energy associated with the biases.Now, since a Hopfield network minimizes its energy, it is hoped that thebinary or continuous Hopfield network’s dynamics will take the state to aminimum that is a valid tour and which might be an optimal tour. This hopeis not fulfilled for large travelling salesman problems, however, without somecareful modifications. We have not specified the size of the weights that enforcethe tour’s validity, relative to the size of the distance weights, and setting thisscale factor poses difficulties. If ‘large’ validity-enforcing weights are used,the network’s dynamics will rattle into a valid state with little regard for thedistances. If ‘small’ validity-enforcing weights are used, it is possible that thedistance weights will cause the network to adopt an invalid state that has lowerenergy than any valid state. Our original formulation of the energy functionputs the objective function and the solution’s validity in potential conflictwith each other. This difficulty has been resolved by the work of Sree Aiyer(1991), who showed how to modify the distance weights so that they would notinterfere with the solution’s validity, and how to define a continuous Hopfieldnetwork whose dynamics are at all times confined to a ‘valid subspace’. Aiyerused a graduated non-convexity or deterministic annealing approach to findgood solutions using these Hopfield networks. The deterministic annealingapproach involves gradually increasing the gain β of the neurons in the networkfrom 0 to ∞, at which point the state of the network corresponds to a validtour. A sequence of trajectories generated by applying this method to a thirtycitytravelling salesman problem is shown in figure 42.11a.A solution to the ‘travelling scholar problem’ found by Aiyer using a continuousHopfield network is shown in figure 42.11b.