Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.41.5: Implementing inference with Gaussian approximations 503tion (41.30) to make predictions. Comparing these predictions with those ofthe Langevin Monte Carlo method (figure 41.7) we observe that, whilst qualitativelythe same, the two are clearly numerically different. So at least one ofthe two methods is not completely accurate.⊲ Exercise 41.3. [2 ] Is the Gaussian approximation to P (w | D, α) too heavy-tailedor too light-tailed, or both? It may help to consider P (w | D, α) as afunction of one parameter w i and to think of the two distributions ona logarithmic scale. Discuss the conditions under which the Gaussianapproximation is most accurate.Why marginalize?If the output is immediately used to make a (0/1) decision and the costs associatedwith error are symmetrical, then the use of marginalized outputs underthis Gaussian approximation will make no difference to the performance of theclassifier, compared with using the outputs given by the most probable parameters,since both functions pass through 0.5 at a MP = 0. But these Bayesianoutputs will make a difference if, for example, there is an option of saying ‘Idon’t know’, in addition to saying ‘I guess 0’ and ‘I guess 1’. And even ifthere are just the two choices ‘0’ and ‘1’, if the costs associated with error areunequal, then the decision boundary will be some contour other than the 0.5contour, and the boundary will be affected by marginalization.