Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.494 41 — Learning as InferenceData set Likelihood Probability of parameters5N = 0(constant)50 w20-5w2-5 0w1 5-5-5 0 5w110N = 25x250-5-10-10 -5 0 5 10x10.5-5 0w1 550 w2-5-5 0w1 550 w2-5-5 0 5w10-5w2x210N = 450-5-10-10 -5 0 5 10x10.10.05-5 0w1 550 w2-5-5 0w1 550 w2-5-5 0 5w150-5w210N = 65x250.050w20-5-10-10 -5 0 5 10x1-5 0w1 550 w2-5-5 0w1 550 w2-5-5 0 5w1-5Figure 41.1. The Bayesian interpretation and generalization of traditional neural network learning.Evolution of the probability distribution over parameters as data arrive.10w2w MPAA5Samples fromP(w|D,H)21(a)B(b)0 5 10B0(c)w1Figure 41.2. Making predictions. (a) The function performed by an optimized neuron w MP (shown bythree of its contours) trained with weight decay, α = 0.01 (from figure 39.6). The contoursshown are those corresponding to a = 0, ±1, namely y = 0.5, 0.27 and 0.73. (b) Are thesepredictions more reasonable? (Contours shown are for y = 0.5, 0.27, 0.73, 0.12 and 0.88.)(c) The posterior probability of w (schematic); the Bayesian predictions shown in (b) wereobtained by averaging together the predictions made by each possible value of the weightsw, with each value of w receiving a vote proportional to its probability under the posteriorensemble. The method used to create (b) is described in section 41.4.