Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.About Chapter 14In this chapter we will draw together several ideas that we’ve encounteredso far in one nice short proof. We will simultaneously prove both Shannon’snoisy-channel coding theorem (for symmetric binary channels) and his sourcecoding theorem (for binary sources). While this proof has connections to manypreceding chapters in the book, it’s not essential to have read them all.On the noisy-channel coding side, our proof will be more constructive thanthe proof given in Chapter 10; there, we proved that almost any random codeis ‘very good’. Here we will show that almost any linear code is very good. Wewill make use of the idea of typical sets (Chapters 4 and 10), and we’ll borrowfrom the previous chapter’s calculation of the weight enumerator function ofrandom linear codes (section 13.5).On the source coding side, our proof will show that random linear hashfunctions can be used for compression of compressible binary sources, thusgiving a link to Chapter 12.228