Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.15 — Further Exercises on Information Theory 239would be a catastrophic failure to communicate information reliably (solid linein figure 15.6).A conservative approach would design the encoding system for the worstcasescenario, installing a code with rate R B ≃ C B (dashed line in figure 15.6).In the event that the lower noise level, f A , holds true, the managers wouldhave a feeling of regret because of the wasted capacity difference C A − R B .Is it possible to create a system that not only transmits reliably at somerate R 0 whatever the noise level, but also communicates some extra, ‘lowerpriority’bits if the noise level is low, as shown in figure 15.7? This codecommunicates the high-priority bits reliably at all noise levels between f A andf B , and communicates the low-priority bits also if the noise level is f A orbelow.This problem is mathematically equivalent to the previous problem, thedegraded broadcast channel. The lower rate of communication was there calledR 0 , and the rate at which the low-priority bits are communicated if the noiselevel is low was called R A .An illustrative answer is shown in figure 15.8, for the case f A = 0.01 andf B = 0.1. (This figure also shows the achievable region for a broadcast channelwhose two half-channels have noise levels f A = 0.01 and f B = 0.1.) I admit Ifind the gap between the simple time-sharing solution and the cunning solutiondisappointingly small.In Chapter 50 we will discuss codes for a special class of broadcast channels,namely erasure channels, where every symbol is either received without erroror erased. These codes have the nice property that they are rateless – thenumber of symbols transmitted is determined on the fly such that reliablecomunication is achieved, whatever the erasure statistics of the channel.Exercise 15.21. [3 ] Multiterminal information networks are both important practicallyand intriguing theoretically. Consider the following example of a two-waybinary channel (figure 15.9a,b): two people both wish to talk over the channel,and they both want to hear what the other person is saying; but you can hearthe signal transmitted by the other person only if you are transmitting a zero.What simultaneous information rates from A to B and from B to A can beachieved, and how? Everyday examples of such networks include the VHFchannels used by ships, and computer ethernet networks (in which all thedevices are unable to hear anything if two or more devices are broadcastingsimultaneously).Obviously, we can achieve rates of 1/ 2 in both directions by simple timesharing.But can the two information rates be made larger? Finding thecapacity of a general two-way channel is still an open problem. However,we can obtain interesting results concerning achievable points for the simplebinary channel discussed above, as indicated in figure 15.9c. There exist codesthat can achieve rates up to the boundary shown. There may exist bettercodes too.CAC BRf AFigure 15.7. Rate of reliablecommunication R, as a function ofnoise level f, for a desiredvariable-rate code.0.60.40.20f B0 0.2 0.4 0.6 0.8 1Figure 15.8. An achievable regionfor the channel with unknownnoise level. Assuming the twopossible noise levels are f A = 0.01and f B = 0.1, the dashed linesshow the rates R A , R B that areachievable using a simpletime-sharing approach, and thesolid line shows rates achievableusing a more cunning approach.fSolutionsSolution to exercise 15.12 (p.235). C(Q) = 5 bits.Hint for the last part: a solution exists that involves a simple (8, 5) code.