Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.212 13 — Binary CodesFor large N, we can use log ( Nw)≃ NH2 (w/N) and R ≃ 1 − M/N to writelog 2 〈A(w)〉 ≃ NH 2 (w/N) − M (13.13)≃ N[H 2 (w/N) − (1 − R)] for any w > 0. (13.14)As a concrete example, figure 13.8 shows the expected weight enumeratorfunction of a rate-1/3 random linear code with N = 540 and M = 360.Gilbert–Varshamov distanceFor weights w such that H 2 (w/N) < (1 − R), the expectation of A(w) issmaller than 1; for weights such that H 2 (w/N) > (1 − R), the expectation isgreater than 1. We thus expect, for large N, that the minimum distance of arandom linear code will be close to the distance d GV defined byH 2 (d GV /N) = (1 − R). (13.15)Definition. This distance, d GV ≡ NH2 −1 (1 − R), is the Gilbert–Varshamovdistance for rate R and blocklength N.The Gilbert–Varshamov conjecture, widely believed, asserts that (for largeN) it is not possible to create binary codes with minimum distance significantlygreater than d GV .Definition. The Gilbert–Varshamov rate R GV is the maximum rate at whichyou can reliably communicate with a bounded-distance decoder (as defined onp.207), assuming that the Gilbert–Varshamov conjecture is true.Why sphere-packing is a bad perspective, and an obsession with distanceis inappropriateIf one uses a bounded-distance decoder, the maximum tolerable noise levelwill flip a fraction f bd = 1 2 d min/N of the bits. So, assuming d min is equal tothe Gilbert distance d GV (13.15), we have:H 2 (2f bd ) = (1 − R GV ). (13.16)R GV = 1 − H 2 (2f bd ). (13.17)Now, here’s the crunch: what did Shannon say is achievable?maximum possible rate of communication is the capacity,He said the6e+525e+524e+523e+522e+521e+5200 100 200 300 400 5001e+601e+401e+2011e-201e-401e-601e-801e-1001e-1200 100 200 300 400 500Figure 13.8. The expected weightenumerator function 〈A(w)〉 of arandom linear code with N = 540and M = 360. Lower figure shows〈A(w)〉 on a logarithmic scale.10.50CapacityR_GV0 0.25 0.5fFigure 13.9. Contrast betweenShannon’s channel capacity C andthe Gilbert rate R GV – themaximum communication rateachievable using abounded-distance decoder, as afunction of noise level f. For anygiven rate, R, the maximumtolerable noise level for Shannonis twice as big as the maximumtolerable noise level for a‘worst-case-ist’ who uses abounded-distance decoder.C = 1 − H 2 (f). (13.18)So for a given rate R, the maximum tolerable noise level, according to Shannon,is given byH 2 (f) = (1 − R). (13.19)Our conclusion: imagine a good code of rate R has been chosen; equations(13.16) and (13.19) respectively define the maximum noise levels tolerable bya bounded-distance decoder, f bd , and by Shannon’s decoder, f.f bd = f/2. (13.20)Bounded-distance decoders can only ever cope with half the noise-level thatShannon proved is tolerable!How does this relate to perfect codes? A code is perfect if there are t-spheres around its codewords that fill Hamming space without overlapping.