Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.13.3: Perfect codes 2091 21 2t. . .t. . .1 21 2t. . .t. . .Figure 13.5. Schematic picture ofHamming space not perfectlyfilled by t-spheres centred on thecodewords of a code. The greyregions show points that are at aHamming distance of more than tfrom any codeword. This is amisleading picture, as, for anycode with large t in highdimensions, the grey spacebetween the spheres takes upalmost all of Hamming space.How happy we would be to use perfect codesIf there were large numbers of perfect codes to choose from, with a widerange of blocklengths and rates, then these would be the perfect solution toShannon’s problem. We could communicate over a binary symmetric channelwith noise level f, for example, by picking a perfect t-error-correcting codewith blocklength N and t = f ∗ N, where f ∗ = f + δ and N and δ are chosensuch that the probability that the noise flips more than t bits is satisfactorilysmall.However, there are almost no perfect codes. The only nontrivial perfectbinary codes are∗1. the Hamming codes, which are perfect codes with t = 1 and blocklengthN = 2 M − 1, defined below; the rate of a Hamming code approaches 1as its blocklength N increases;2. the repetition codes of odd blocklength N, which are perfect codes witht = (N − 1)/2; the rate of repetition codes goes to zero as 1/N; and3. one remarkable 3-error-correcting code with 2 12 codewords of blocklengthN = 23 known as the binary Golay code. [A second 2-errorcorrectingGolay code of length N = 11 over a ternary alphabet was discoveredby a Finnish football-pool enthusiast called Juhani Virtakallioin 1947.]There are no other binary perfect codes. Why this shortage of perfect codes?Is it because precise numerological coincidences like those satisfied by theparameters of the Hamming code (13.4) and the Golay code,( ) ( ) ( )23 23 231 + + + = 2 11 , (13.5)1 2 3are rare? Are there plenty of ‘almost-perfect’ codes for which the t-spheres fillalmost the whole space?No. In fact, the picture of Hamming spheres centred on the codewordsalmost filling Hamming space (figure 13.5) is a misleading one: for most codes,whether they are good codes or bad codes, almost all the Hamming space istaken up by the space between t-spheres (which is shown in grey in figure 13.5).Having established this gloomy picture, we spend a moment filling in theproperties of the perfect codes mentioned above.