Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.29.2: Importance sampling 361Entropy64 log(2)Figure 29.4. (a) Entropy of a64-spin Ising model as a functionof temperature. (b) One state of a1024-spin Ising model.(a)00 1 2 3 4 5 6Temperature(b)The number of samples required to hit the typical set once is thus of orderR min ≃ 2 N−H . (29.18)So, what is H? At high temperatures, the probability distribution of an Isingmodel tends to a uniform distribution and the entropy tends to H max = Nbits, which means R min is of order 1. Under these conditions, uniform samplingmay well be a satisfactory technique for estimating Φ. But high temperaturesare not of great interest. Considerably more interesting are intermediate temperaturessuch as the critical temperature at which the Ising model melts froman ordered phase to a disordered phase. The critical temperature of an infiniteIsing model, at which it melts, is θ c = 2.27. At this temperature the entropyof an Ising model is roughly N/2 bits (figure 29.4). For this probability distributionthe number of samples required simply to hit the typical set once isof orderR min ≃ 2 N−N/2 = 2 N/2 , (29.19)which for N = 1000 is about 10 150 . This is roughly the square of the numberof particles in the universe. Thus uniform sampling is utterly useless for thestudy of Ising models of modest size. And in most high-dimensional problems,if the distribution P (x) is not actually uniform, uniform sampling is unlikelyto be useful.OverviewHaving established that drawing samples from a high-dimensional distributionP (x) = P ∗ (x)/Z is difficult even if P ∗ (x) is easy to evaluate, we will nowstudy a sequence of more sophisticated Monte Carlo methods: importancesampling, rejection sampling, the Metropolis method, Gibbs sampling, andslice sampling.29.2 Importance samplingImportance sampling is not a method for generating samples from P (x) (problem1); it is just a method for estimating the expectation of a function φ(x)(problem 2). It can be viewed as a generalization of the uniform samplingmethod.For illustrative purposes, let us imagine that the target distribution is aone-dimensional density P (x). Let us assume that we are able to evaluate thisdensity at any chosen point x, at least to within a multiplicative constant;thus we can evaluate a function P ∗ (x) such thatP (x) = P ∗ (x)/Z. (29.20)