Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.42.9: Hopfield networks for optimization problems 517ACity BCDAPlace in tour1 2 3 4BDACity BCDAPlace in tour1 2 3 4BD(b)ABCDAB112 3 42 3 4Figure 42.10. Hopfield network forsolving a travelling salesmanproblem with K = 4 cities. (a1,2)Two solution states of the16-neuron network, with activitesrepresented by black = 1, white =0; and the tours corresponding tothese network states. (b) Thenegative weights between node B2and other nodes; these weightsenforce validity of a tour. (c) Thenegative weights that embody thedistance objective function.C(a1)C(a2)(c)CD−d BDHopfield and Tank (1985) suggested that one might take an interestingconstraint satisfaction problem and design the weights of a binary or continuousHopfield network such that the settling process of the network wouldminimize the objective function of the problem.The travelling salesman problemA classic constraint satisfaction problem to which Hopfield networks have beenapplied is the travelling salesman problem.A set of K cities is given, and a matrix of the K(K−1)/2 distances betweenthose cities. The task is to find a closed tour of the cities, visiting each cityonce, that has the smallest total distance. The travelling salesman problem isequivalent in difficulty to an NP-complete problem.The method suggested by Hopfield and Tank is to represent a tentative solutionto the problem by the state of a network with I = K 2 neurons arrangedin a square, with each neuron representing the hypothesis that a particularcity comes at a particular point in the tour. It will be convenient to considerthe states of the neurons as being between 0 and 1 rather than −1 and 1.Two solution states for a four-city travelling salesman problem are shown infigure 42.10a.The weights in the Hopfield network play two roles. First, they must definean energy function which is minimized only when the state of the networkrepresents a valid tour. A valid state is one that looks like a permutationmatrix, having exactly one ‘1’ in every row and one ‘1’ in every column. Thisrule can be enforced by putting large negative weights between any pair ofneurons that are in the same row or the same column, and setting a positivebias for all neurons to ensure that K neurons do turn on. Figure 42.10b showsthe negative weights that are connected to one neuron, ‘B2’, which representsthe statement ‘city B comes second in the tour’.Second, the weights must encode the objective function that we wantto minimize – the total distance. This can be done by putting negativeweights proportional to the appropriate distances between the nodes in adjacentcolumns. For example, between the B and D nodes in adjacent columns,the weight would be −d BD . The negative weights that are connected to neuronB2 are shown in figure 42.10c. The result is that when the network is ina valid state, its total energy will be the total distance of the corresponding