Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.39.3: Training the single neuron as a binary classifier 477103wx 282 2.5261.5140.520(a)(c) -0.50-0.5 0 0.5 1 1.5 2 2.5 3w 10 2 4 6 8 10x 1101020-2-4-6-8-10w0w1w28642(f) 00 2 4 6 8 10108642(h) 00 2 4 6 8 10108642(j) 00 2 4 6 8 108642(g) 00 2 4 6 8 10108642(i) 00 2 4 6 8 10108642(k) 00 2 4 6 8 10(b)-121 10 100 1000 10000 1000007G(w)65432(d)(e)101 10 100 1000 10000 100000400350300250200150100E_W(w)5001 10 100 1000 10000 100000Figure 39.4. A single neuron learning to classify by gradient descent. The neuron has two weights w 1and w 2 and a bias w 0 . The learning rate was set to η = 0.01 and batch-mode gradientdescent was performed using the code displayed in algorithm 39.5. (a) The training data.(b) Evolution of weights w 0 , w 1 and w 2 as a function of number of iterations (on log scale).(c) Evolution of weights w 1 and w 2 in weight space. (d) The objective function G(w) as afunction of number of iterations. (e) The magnitude of the weights E W (w) as a function oftime. (f–k) The function performed by the neuron (shown by three of its contours) after 30,80, 500, 3000, 10 000 and 40 000 iterations. The contours shown are those corresponding toa = 0, ±1, namely y = 0.5, 0.27 and 0.73. Also shown is a vector proportional to (w 1 , w 2 ).The larger the weights are, the bigger this vector becomes, and the closer together are thecontours.