Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.42 2 — Probability, Entropy, and Inferencesof(a)1≡1 + exp(−a) = 1 ( )1 − e−a2 1 + e −a + 1 ()= 1 e a/2 − e −a/22 e a/2 + e −a/2 + 1 = 1 (tanh(a/2) + 1). (2.67)2In the case b = log 2 p/q, we can repeat steps (2.63–2.65), replacing e by 2,to obtain1p = . (2.68)1 + 2−b Solution to exercise 2.18 (p.36).P (y | x)P (x)P (x | y) = (2.69)P (y)P (x = 1 | y) P (y | x = 1) P (x = 1)⇒ = (2.70)P (x = 0 | y) P (y | x = 0) P (x = 0)P (x = 1 | y) P (y | x = 1) P (x = 1)⇒ log = log + logP (x = 0 | y) P (y | x = 0) P (x = 0) . (2.71)Solution to exercise 2.19 (p.36). The conditional independence of d 1 and d 2given x meansP (x, d 1 , d 2 ) = P (x)P (d 1 | x)P (d 2 | x). (2.72)This gives a separation of the posterior probability ratio into a series of factors,one for each data point, times the prior probability ratio.P (x = 1 | {d i })P (x = 0 | {d i })= P ({d i} | x = 1) P (x = 1)P ({d i } | x = 0) P (x = 0)(2.73)= P (d 1 | x = 1) P (d 2 | x = 1) P (x = 1)P (d 1 | x = 0) P (d 2 | x = 0) P (x = 0) . (2.74)Life in high-dimensional spacesSolution to exercise 2.20 (p.37).N dimensions is in factV (r, N) =The volume of a hypersphere of radius r inπN/2(N/2)! rN , (2.75)but you don’t need to know this. For this question all that we need is ther-dependence, V (r, N) ∝ r N . So the fractional volume in (r − ɛ, r) isr N − (r − ɛ) Nr N(= 1 − 1 − ɛ ) N. (2.76)rThe fractional volumes in the shells for the required cases are:N 2 10 1000ɛ/r = 0.01 0.02 0.096 0.99996ɛ/r = 0.5 0.75 0.999 1 − 2 −1000Notice that no matter how small ɛ is, for large enough N essentially all theprobability mass is in the surface shell of thickness ɛ.