Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.Index 621Bob, 199Boltzmann entropy, 85Boltzmann machine, 522bombes, 265book ISBN, 235bookies, 456Bottou, Leon, 121bound, 85union, 166, 216, 230bounded-distance decoder, 207, 212bounding chain, 419box, 343, 351boyish matters, 58brain, 468Bridge, 126British, 260broadcast channel, 237, 239, 594Brody, Carlos, 246Brownian motion, 280, 316, 535BSC, see channel, binary symmetricbudget, 94, 96Buffon’s needle, 38BUGS, 371, 431buoy, 307burglar alarm and earthquake, 293Burrows–Wheeler transform, 121burst errors, 185, 186bus-stop paradox, 39, 46, 107byte, 134, 265cable labelling, 175calculator, 320camera, 549canonical, 88capacity, 14, 146, 150, 151, 183, 484channel with synchronizationerrors, 187constrained channel, 251Gaussian channel, 182Hopfield network, 514neural network, 483neuron, 483symmetry argument, 151car data reception, 594card, 233casting out nines, 198Cauchy distribution, 85, 88, 313, 362caution, see sermonequipartition, 83Gaussian distribution, 312importance sampling, 362, 382sampling theory, 64cave, 214caveat, see caution and sermoncellphone, see mobile phonecellular automaton, 130central-limit theorem, 36, 41, 88, 131,see law of large numberscentre of gravity, 35chain rule, 528challenges, 246Tanner, 569channelAWGN, 177binary erasure, 148, 151binary symmetric, 4, 146, 148,151, 206, 211, 215, 229broadcast, 237, 239, 594bursty, 185, 557capacity, 14, 146, 150, 250connection with physics, 257coding theorem, seenoisy-channel codingtheoremcomplex, 184, 557constrained, 248, 255, 256continuous, 178discrete memoryless, 147erasure, 188, 219, 589extended, 153fading, 186Gaussian, 155, 177, 186input ensemble, 150multiple access, 237multiterminal, 239noiseless, 248noisy, 3, 146noisy typewriter, 148, 152symmetric, 171two-dimensional, 262unknown noise level, 238variable symbol durations, 256with dependent sources, 236with memory, 557Z channel, 148, 149, 150, 172cheat, 200Chebyshev inequality, 81, 85checkerboard, 404, 520Chernoff bound, 85chess, 451chess board, 406, 520chi-squared, 27, 40, 323, 458Cholesky decomposition, 552chromatic aberration, 552cinema, 187circle, 316classical statistics, 64criticisms, 32, 50, 457classifier, 532Clockville, 39clustering, 284, 303coalescence, 413cocked hat, 307code, see error-correcting code, sourcecode (for data compression),symbol code, arithmeticcoding, linear code, randomcode or hash codedual, see error-correcting code,dualfor constrained channel, 249variable-length, 255code-equivalent, 576codebreakers, 265codeword, see source code, symbolcode, or error-correctingcodecoding theory, 4, 19, 205, 215, 574coin, 1, 30, 38, 63, 76, 307, 464coincidence, 267, 343, 351collective, 403collision, 200coloured noise, 179combination, 2, 490, 598commander, 241communication, v, 3, 16, 138, 146,156, 162, 167, 178, 182, 186,192, 205, 210, 215, 394, 556,562, 596broadcast, 237of dependent information, 236over noiseless channels, 248perspective on learning, 483, 512competitive learning, 285complexity, 531, 548complexity control, 289, 346, 347, 349compress, 119compression, see source codefuture methods, 129lossless, 74lossy, 74, 284, 285of already-compressed files, 74of any file, 74universal, 121computer, 370concatenation, 185, 214, 220error-correcting codes, 16, 21,184, 185, 579in compression, 92in Markov chains, 373concave ⌢, 35conditional entropy, 138, 146cones, 554confidence interval, 457, 464confidence level, 464confused gameshow host, 57conjugate gradient, 479conjugate prior, 319conjuror, 233connection betweenchannel capacity and physics, 257error correcting code and latentvariable model, 437pattern recognition anderror-correction, 481supervised and unsupervisedlearning, 515vector quantization anderror-correction, 285connection matrix, 253, 257constrained channel, 248, 257, 260,399variable-length code, 249constraint satisfaction, 516content-addressable memory, 192, 193,469, 505continuous channel, 178control treatment, 458conventions, see notationconvex hull, 102convex ⌣, 35convexity, 370convolution, 568convolutional code, 184, 186, 574, 587equivalence, 576