Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.70 4 — The Source Coding Theoremso a strategy that uses such outcomes must sometimes take longer to find theright answer.The insight that the outcomes should be as near as possible to equiprobablemakes it easier to search for an optimal strategy. The first weighing mustdivide the 24 possible hypotheses into three groups of eight. Then the secondweighing must be chosen so that there is a 3:3:2 split of the hypotheses.Thus we might conclude:the outcome of a random experiment is guaranteed to be most informativeif the probability distribution over outcomes is uniform.This conclusion agrees with the property of the entropy that you provedwhen you solved exercise 2.25 (p.37): the entropy of an ensemble X is biggestif all the outcomes have equal probability p i = 1/|A X |.Guessing gamesIn the game of twenty questions, one player thinks of an object, and theother player attempts to guess what the object is by asking questions thathave yes/no answers, for example, ‘is it alive?’, or ‘is it human?’ The aimis to identify the object with as few questions as possible. What is the beststrategy for playing this game? For simplicity, imagine that we are playingthe rather dull version of twenty questions called ‘sixty-three’.Example 4.3. The game ‘sixty-three’. What’s the smallest number of yes/noquestions needed to identify an integer x between 0 and 63?Intuitively, the best questions successively divide the 64 possibilities into equalsized sets. Six questions suffice. One reasonable strategy asks the followingquestions:1: is x ≥ 32?2: is x mod 32 ≥ 16?3: is x mod 16 ≥ 8?4: is x mod 8 ≥ 4?5: is x mod 4 ≥ 2?6: is x mod 2 = 1?[The notation x mod 32, pronounced ‘x modulo 32’, denotes the remainderwhen x is divided by 32; for example, 35 mod 32 = 3 and 32 mod 32 = 0.]The answers to these questions, if translated from {yes, no} to {1, 0}, givethe binary expansion of x, for example 35 ⇒ 100011.✷What are the Shannon information contents of the outcomes in this example?If we assume that all values of x are equally likely, then the answersto the questions are independent and each has Shannon information contentlog 2 (1/0.5) = 1 bit; the total Shannon information gained is always six bits.Furthermore, the number x that we learn from these questions is a six-bit binarynumber. Our questioning strategy defines a way of encoding the randomvariable x as a binary file.So far, the Shannon information content makes sense: it measures thelength of a binary file that encodes x. However, we have not yet studiedensembles where the outcomes have unequal probabilities. Does the Shannoninformation content make sense there too?