Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.5.9: Solutions 10501000110110000010100111001011101110000000100100011010001010110011110001001101010111100110111101111The total symbol code budgetFigure 5.8. The codewordsupermarket and the symbolcoding budget. The ‘cost’ 2 −l ofeach codeword (with length l) isindicated by the size of the box itis written in. The total budgetavailable when making a uniquelydecodeable code is 1.symbol probability Huffman Rival code’s Modified rivalcodewords codewords codea p a c H (a) c R (a) c R (c)b p b c H (b) c R (b) c R (b)c p c c H (c) c R (c) c R (a)top, and purchase the first codeword of the required length. We advancedown the supermarket a distance 2 −l , and purchase the next codeword of thenext required length, and so forth. Because the codeword lengths are gettinglonger, and the corresponding intervals are getting shorter, we can alwaysbuy an adjacent codeword to the latest purchase, so there is no wasting ofthe budget. Thus at the Ith codeword we have advanced a distance ∑ Ii=1 2−l idown the supermarket; if ∑ 2 −l i≤ 1, we will have purchased all the codewordswithout running out of budget.Figure 5.9. Proof that Huffmancoding makes an optimal symbolcode. We assume that the rivalcode, which is said to be optimal,assigns unequal length codewordsto the two symbols with smallestprobability, a and b. Byinterchanging codewords a and cof the rival code, where c is asymbol with rival codelength aslong as b’s, we can make a codebetter than the rival code. Thisshows that the rival code was notoptimal.Solution to exercise 5.16 (p.99). The proof that Huffman coding is optimaldepends on proving that the key step in the algorithm – the decision to givethe two symbols with smallest probability equal encoded lengths – cannotlead to a larger expected length than any other code. We can prove this bycontradiction.Assume that the two symbols with smallest probability, called a and b,to which the Huffman algorithm would assign equal length codewords, do nothave equal lengths in any optimal symbol code. The optimal symbol codeis some other rival code in which these two codewords have unequal lengthsl a and l b with l a < l b . Without loss of generality we can assume that thisother code is a complete prefix code, because any codelengths of a uniquelydecodeable code can be realized by a prefix code.In this rival code, there must be some other symbol c whose probabilityp c is greater than p a and whose length in the rival code is greater than orequal to l b , because the code for b must have an adjacent codeword of equalor greater length – a complete prefix code never has a solo codeword of themaximum length.Consider exchanging the codewords of a and c (figure 5.9), so that a is