Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.Index 625life in high dimensions, 37, 124likelihood, 6, 28, 49, 152, 324, 529, 558contrasted with probability, 28subjectivity, 30likelihood equivalence, 447likelihood principle, 32, 61, 464limit cycle, 508linear block code, 9, 11, 19, 171, 183,186, 206, 229coding theorem, 229decoding, 184linear regression, 342, 527linear-feedback shift-register, 184, 574Litsyn, Simon, 572little ’n’ large data set, 288log-normal, 315logarithms, 2logit, 307, 316long thin strip, 409loopy belief propagation, 434loopy message-passing, 338, 340, 556loss function, 451lossy compression, 168, 284, 285low-density generator-matrix code,207, 590low-density parity-check code, 557,see error-correcting codeLT code, 590Luby, Michael G., 568, 590Luria, Salvador, 446Lyapunov function, 287, 291, 508,520, 521machine learning, 246macho, 319MacKay, David J.C., 187, 496, 557magician, 233magnet, 602magnetic recording, 593majority vote, 5male, 277Mandelbrot, Benoit, 262MAP, see maximum a posteriorimapping, 92marginal entropy, 139, 140marginal likelihood, 29, 298, 322, seeevidencemarginal probability, 23, 147marginalization, 29, 295, 319Markov chain, 141, 168construction, 373Markov chain Monte Carlo, see MonteCarlo methodsMarkov model, 111, 437, see Markovchainmarriage, 454matrix, 409matrix identities, 438max–product, 339maxent, 308, see maximum entropymaximum distance separable, 219maximum entropy, 308, 551maximum likelihood, 6, 152, 300, 347maximum a posteriori, 6, 307, 325,538McCollough effect, 553MCMC (Markov chain Monte Carlo),see Monte Carlo methodsMcMillan, B., 95MD5, 200MDL, see minimum description lengthMDS, 220mean, 1mean field theory, 422, 425melody, 201, 203memory, 468address-based, 468associative, 468, 505content-addressable, 192, 469MemSys, 551message passing, 187, 241, 248, 283,324, 407, 556, 591BCJR, 330belief propagation, 330forward–backward, 330in graphs with cycles, 338loopy, 338, 340, 434sum–product algorithm, 336Viterbi, 329metacode, 104, 108metric, 512Metropolis method, 496, see MonteCarlo methodsMézard, Marc, 340micro-saccades, 554microcanonical, 87microsoftus, 458microwave oven, 127min–sum algorithm, 245, 325, 329,339, 578, 581mine (hole in ground), 451minimax, 455minimization, 473, see optimizationminimum description length, 352minimum distance, 206, 214, seedistanceMinka, Thomas, 340mirror, 529Mitzenmacher, Michael, 568mixing coefficients, 298, 312mixture modelling, 282, 284, 303, 437mixture of Gaussians, 312mixtures in Markov chains, 373ML, see maximum likelihoodMLP, see multilayer perceptronMML, see minimum descriptionlengthmobile phone, 182, 186model, 111, 120model comparison, 198, 346, 347, 349typical evidence, 54, 60modelling, 285density modelling, 284, 303images, 524latent variable models, 353, 432,437nonparametric, 538moderation, 29, 498, seemarginalizationmolecules, 201Molesworth, 241momentum, 387, 479Monte Carlo methods, 357, 498acceptance rate, 394acceptance ratio method, 379and communication, 394annealed importance sampling,379coalescence, 413dependence on dimension, 358exact sampling, 413for visualization, 551Gibbs sampling, 370, 391, 418Hamiltonian Monte Carlo, 387,496hybrid Monte Carlo, seeHamiltonian Monte Carloimportance sampling, 361, 379weakness of, 382Langevin method, 498Markov chain Monte Carlo, 365Metropolis method, 365dumb Metropolis, 394, 496Metropolis–Hastings, 365multi-state, 392, 395, 398overrelaxation, 390, 391perfect simulation, 413random walk suppression, 370,387random-walk Metropolis, 388rejection sampling, 364adaptive, 370reversible jump, 379simulated annealing, 379, 392slice sampling, 374thermodynamic integration, 379umbrella sampling, 379Monty Hall problem, 57Morse, 256motorcycle, 110movie, 551multilayer perceptron, 529, 535multiple access channel, 237multiterminal networks, 239multivariate Gaussian, 176Munro–Robbins theorem, 441murder, 26, 58, 61, 354music, 201, 203mutation rate, 446mutual information, 139, 146, 150, 151how to compute, 149myth, 347compression, 74nat (unit), 264, 601natural gradient, 443natural selection, 269navigation, 594Neal, Radford M., 111, 121, 187, 374,379, 391, 392, 397, 419, 420,429, 432, 496needle, Buffon’s, 38network, 529neural network, 468, 470capacity, 483learning as communication, 483learning as inference, 492