Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.29.4: The Metropolis–Hastings method 367ɛP ∗ (x)Figure 29.11. Metropolis methodin two dimensions, showing atraditional proposal density thathas a sufficiently small step size ɛthat the acceptance frequency willbe about 0.5.Q(x; x (1) )x (1)Lwith a length scale ɛ that is short relative to the longest length scale L of theprobable region (figure 29.11). A reason for choosing a small length scale isthat for most high-dimensional problems, a large random step from a typicalpoint (that is, a sample from P (x)) is very likely to end in a state that hasvery low probability; such steps are unlikely to be accepted. If ɛ is large,movement around the state space will only occur when such a transition to alow-probability state is actually accepted, or when a large random step chancesto land in another probable state. So the rate of progress will be slow if largesteps are used.The disadvantage of small steps, on the other hand, is that the Metropolismethod will explore the probability distribution by a random walk, and arandom walk takes a long time to get anywhere, especially if the walk is madeof small steps.Exercise 29.3. [1 ] Consider a one-dimensional random walk, on each step ofwhich the state moves randomly to the left or to the right with equalprobability. Show that after T steps of size ɛ, the state is likely to havemoved only a distance about √ T ɛ. (Compute the root mean squaredistance travelled.)Recall that the first aim of Monte Carlo sampling is to generate a number ofindependent samples from the given distribution (a dozen, say). If the largestlength scale of the state space is L, then we have to simulate a random-walkMetropolis method for a time T ≃ (L/ɛ) 2 before we can expect to get a samplethat is roughly independent of the initial condition – and that’s assuming thatevery step is accepted: if only a fraction f of the steps are accepted on average,then this time is increased by a factor 1/f.Rule of thumb: lower bound on number of iterations of aMetropolis method. If the largest length scale of the space ofprobable states is L, a Metropolis method whose proposal distributiongenerates a random walk with step size ɛ must be run for atleastT ≃ (L/ɛ) 2 (29.32)iterations to obtain an independent sample.This rule of thumb gives only a lower bound; the situation may be muchworse, if, for example, the probability distribution consists of several islandsof high probability separated by regions of low probability.