Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.ContentsPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v1 Introduction to Information Theory . . . . . . . . . . . . . 32 Probability, Entropy, and Inference . . . . . . . . . . . . . . 223 More about Inference . . . . . . . . . . . . . . . . . . . . . 48I Data Compression . . . . . . . . . . . . . . . . . . . . . . 654 The Source Coding Theorem . . . . . . . . . . . . . . . . . 675 Symbol Codes . . . . . . . . . . . . . . . . . . . . . . . . . 916 Stream Codes . . . . . . . . . . . . . . . . . . . . . . . . . . 1107 Codes for Integers . . . . . . . . . . . . . . . . . . . . . . . 132II Noisy-Channel Coding . . . . . . . . . . . . . . . . . . . . 1378 Dependent Random Variables . . . . . . . . . . . . . . . . . 1389 Communication over a Noisy Channel . . . . . . . . . . . . 14610 The Noisy-Channel Coding Theorem . . . . . . . . . . . . . 16211 Error-Correcting Codes and Real Channels . . . . . . . . . 177III Further Topics in Information Theory . . . . . . . . . . . . . 19112 Hash Codes: Codes for Efficient Information Retrieval . . 19313 Binary Codes . . . . . . . . . . . . . . . . . . . . . . . . . 20614 Very Good Linear Codes Exist . . . . . . . . . . . . . . . . 22915 Further Exercises on Information Theory . . . . . . . . . . 23316 Message Passing . . . . . . . . . . . . . . . . . . . . . . . . 24117 Communication over Constrained Noiseless Channels . . . 24818 Crosswords and Codebreaking . . . . . . . . . . . . . . . . 26019 Why have Sex? Information Acquisition and Evolution . . 269IV Probabilities and Inference . . . . . . . . . . . . . . . . . . 28120 An Example Inference Task: Clustering . . . . . . . . . . . 28421 Exact Inference by Complete Enumeration . . . . . . . . . 29322 Maximum Likelihood and Clustering . . . . . . . . . . . . . 30023 Useful Probability Distributions . . . . . . . . . . . . . . . 31124 Exact Marginalization . . . . . . . . . . . . . . . . . . . . . 31925 Exact Marginalization in Trellises . . . . . . . . . . . . . . 32426 Exact Marginalization in Graphs . . . . . . . . . . . . . . . 33427 Laplace’s Method . . . . . . . . . . . . . . . . . . . . . . . 341