Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.15 — Further Exercises on Information Theory 235Exercise 15.12. [3, p.239] The input to a channel Q is a word of 8 bits. Theoutput is also a word of 8 bits. Each time it is used, the channel flipsexactly one of the transmitted bits, but the receiver does not know whichone. The other seven bits are received without error. All 8 bits areequally likely to be the one that is flipped. Derive the capacity of thischannel.Show, by describing an explicit encoder and decoder that it is possiblereliably (that is, with zero error probability) to communicate 5 bits percycle over this channel.⊲ Exercise 15.13. [2 ] A channel with input x ∈ {a, b, c} and output y ∈ {r, s, t, u}has conditional probability matrix:Q =⎡⎢⎣What is its capacity?⎤1/ 2 0 01/ 2 1/ 2 00 1/ 2 1/ 20 0 1/ 2⎥⎦ .ra✟✟✯❍❍❥ sb✟✟✯❍❍❥c✟✟✯t❍❍❥ u⊲ Exercise 15.14. [3 ] The ten-digit number on the cover of a book known as theISBN incorporates an error-detecting code. The number consists of ninesource digits x 1 , x 2 , . . . , x 9 , satisfying x n ∈ {0, 1, . . . , 9}, and a tenthcheck digit whose value is given by( 9∑)x 10 = nx n mod 11.n=10-521-64298-11-010-00000-4Table 15.1. Some valid ISBNs.[The hyphens are included forlegibility.]Here x 10 ∈ {0, 1, . . . , 9, 10}. If x 10 = 10 then the tenth digit is shownusing the roman numeral X.Show that a valid ISBN satisfies:( 10∑n=1nx n)mod 11 = 0.Imagine that an ISBN is communicated over an unreliable human channelwhich sometimes modifies digits and sometimes reorders digits.Show that this code can be used to detect (but not correct) all errors inwhich any one of the ten digits is modified (for example, 1-010-00000-4→ 1-010-00080-4).Show that this code can be used to detect all errors in which any two adjacentdigits are transposed (for example, 1-010-00000-4 → 1-100-00000-4).What other transpositions of pairs of non-adjacent digits can be detected?If the tenth digit were defined to be( 9∑)nx nn=1x 10 =mod 10,why would the code not work so well? (Discuss the detection of bothmodifications of single digits and transpositions of digits.)