Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.2.10: Solutions 41Solution to exercise 2.14 (p.35).We wish to prove, given the propertyf(λx 1 + (1 − λ)x 2 ) ≤ λf(x 1 ) + (1 − λ)f(x 2 ), (2.60)that, if ∑ p i = 1 and p i ≥ 0,(I∑I∑)p i f(x i ) ≥ f p i x i . (2.61)i=1i=1We proceed by recursion, working from the right-hand side. (This proof doesnot handle cases where some p i = 0; such details are left to the pedanticreader.) At the first line we use the definition of convexity (2.60) with λ =p 1Ii=1 p i= p 1 ; at the second line, λ = p2I .i=2 p i( I∑) ()I∑f p i x i = f p 1 x 1 + p i x ii=1i=2[ I∑] ( I∑/ I∑)]≤ p 1 f(x 1 ) + p i[f p i x i p ii=2i=2 i=2[ I∑] [p 2≤ p 1 f(x 1 ) + p i ∑ Ii=2 i=2 p f (x 2 ) +i(2.62)∑ ( I I∑/ I∑)]i=3 p i∑ Ii=2 p f p i x i p i ,i i=3 i=3and so forth.✷Solution to exercise 2.16 (p.36).(a) For the outcomes {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}, the probabilities are P ={ 136 , 2 36 , 3 36 , 4 36 , 536 , 636 , 536 , 436 , 336 , 236 , 1 36 }.(b) The value of one die has mean 3.5 and variance 35/12. So the sum ofone hundred has mean 350 and variance 3500/12 ≃ 292, and by thecentral-limit theorem the probability distribution is roughly Gaussian(but confined to the integers), with this mean and variance.(c) In order to obtain a sum that has a uniform distribution we have to startfrom random variables some of which have a spiky distribution with theprobability mass concentrated at the extremes. The unique solution isto have one ordinary die and one with faces 6, 6, 6, 0, 0, 0.(d) Yes, a uniform distribution can be created in several ways, for exampleby labelling the rth die with the numbers {0, 1, 2, 3, 4, 5} × 6 r .Solution to exercise 2.17 (p.36).and q = 1 − p givesThe hyperbolic tangent isa = ln p qp1 − p⇒ p =⇒pq = ea (2.63)= e a (2.64)e ae a + 1 = 11 + exp(−a) . (2.65)tanh(a) = ea − e −ae a + e −a (2.66)To think about: does this uniformdistribution contradict thecentral-limit theorem?