Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.5.2: What limit is imposed by unique decodeability? 95alphabet {0, 1}, the codeword lengths must satisfy:where I = |A X |.I∑2 −l i≤ 1, (5.9)i=1Completeness. If a uniquely decodeable code satisfies the Kraft inequalitywith equality then it is called a complete code.We want codes that are uniquely decodeable; prefix codes are uniquely decodeable,and are easy to decode. So life would be simpler for us if we couldrestrict attention to prefix codes. Fortunately, for any source there is an optimalsymbol code that is also a prefix code.Kraft inequality and prefix codes. Given a set of codeword lengths thatsatisfy the Kraft inequality, there exists a uniquely decodeable prefixcode with these codeword lengths.The Kraft inequality might be more accurately referred to as the Kraft–McMillan inequality: Kraft proved that if the inequality is satisfied, then aprefix code exists with the given lengths. McMillan (1956) proved the converse,that unique decodeability implies that the inequality holds.Proof of the Kraft inequality.Define S = ∑ i 2−l i. Consider the quantityS N =[ ∑iN I∑ I∑ I∑2 i] −l = · · · 2 − (l i 1+ l i2 + · · · l iN ) . (5.10)i 1=1 i 2=1 i N =1The quantity in the exponent, (l i1 + l i2 + · · · + l iN ), is the length of theencoding of the string x = a i1 a i2 . . . a iN . For every string x of length N,there is one term in the above sum. Introduce an array A l that countshow many strings x have encoded length l. Then, defining l min = min i l iand l max = max i l i :S N =Nl∑ maxl=Nl min2 −l A l . (5.11)Now assume C is uniquely decodeable, so that for all x ≠ y, c + (x) ≠c + (y). Concentrate on the x that have encoded length l. There are atotal of 2 l distinct bit strings of length l, so it must be the case thatA l ≤ 2 l . SoS N =Nl∑ maxl=Nl min2 −l A l ≤Nl∑ maxl=Nl min1 ≤ Nl max . (5.12)Thus S N ≤ l max N for all N. Now if S were greater than 1, then as Nincreases, S N would be an exponentially growing function, and for largeenough N, an exponential always exceeds a polynomial such as l max N.But our result (S N ≤ l max N) is true for any N. Therefore S ≤ 1. ✷⊲ Exercise 5.14. [3, p.104] Prove the result stated above, that for any set of codewordlengths {l i } satisfying the Kraft inequality, there is a prefix codehaving those lengths.