Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.436 33 — Variational Methods(a)(b)Figure 33.6. Two separableGaussian approximations (dottedlines) to a bivariate Gaussiandistribution (solid line). (a) Theapproximation that minimizes thevariational free energy. (b) Theapproximation that minimizes theobjective function G. In eachfigure, the lines show the contoursat which x T Ax = 1, where A isthe inverse covariance matrix ofthe Gaussian.In contrast, if we use the objective function G then we find:()G(σ 2 Q ) = 1 2ln σQ 2 + σ2 1σQ2 + ln σQ 2 + σ2 2σQ2+ constant, (33.60)where the constant depends on σ 1 and σ 2 only. Differentiating,[ ( )]dd ln σQ 2 G = 1 σ12 2 −2 σQ2 + σ2 2σQ2 , (33.61)which is zero whenσ 2 Q = 1 2(σ21 + σ22 ). (33.62)Thus we set the approximating distribution’s variance to the mean varianceof the target distribution P .In the case σ 1 = 10 and σ 2 = 1, we obtain σ Q ≃ 10/ √ 2, which is just afactor of √ 2 smaller than σ 1 , independent of the value of σ 2 .The two approximations are shown to scale in figure 33.6.Solution to exercise 33.6 (p.434). The best possible variational approximationis of course the target distribution P . Assuming that this is not possible, agood variational approximation is more compact than the true distribution.In contrast, a good sampler is more heavy tailed than the true distribution.An over-compact distribution would be a lousy sampler with a large variance.