Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.628 Indexspread spectrum, 182, 188spring, 291spy, 464square, 38staircase, 569, 587stalactite, 214standard deviation, 320stars, 307state diagram, 251statistic, 458sufficient, 300statistical physics, see physicsstatistical test, 51, 458steepest descents, 441stereoscopic vision, 524stiffness, 289Stirling’s approximation, 1, 8stochastic, 472stochastic dynamics, see HamiltonianMonte Carlostochastic gradient, 476stop-when-it’s-done, 561, 583stopping rule, 463straws, drawing, 233stream codes, 110–130student, 125Student-t distribution, 312, 323subjective probability, 26, 30submarine, 71subscriber, 593subset, 66substring, 119sufficient statistics, 300sum rule, 39, 46sum–product algorithm, 187, 245, 326,336, 407, 434, 556, 572, 578summary, 335summary state, 418summation convention, 438super-channel, 184supermarket (codewords), 96, 104, 112support vector, 548surprise value, 264survey propagation, 340suspicious coincidences, 351symbol code, 91budget, 94, 96codeword, 92disadvantages, 100optimal, 91self-delimiting, 132supermarket, 112symmetric channel, 171symmetry argument, 151synchronization, 249synchronization errors, 187syndrome, 10, 11, 20syndrome decoding, 11, 216, 229, 371systematic, 575t-distribution, see Student-ttail, 85, 312, 313, 440, 446, 503, 584tamper detection, 199Tank, David W., 517Tanner challenge, 569Tanner product code, 571Tanner, Michael, 569Tanzanite, 451tap, 575telephone, see phonetelescope, 529temperature, 392, 601termination, 579terminology, 598, see notationMonte Carlo methods, 372testfluctuation, 446statistical, 51, 458text entry, 118thermal distribution, 88thermodynamic integration, 379thermodynamics, 404, 601third law, 406Thiele, T.N., 548thin shell, 37, 125third law of thermodynamics, 406Thitimajshima, P., 186three cards, 142three doors, 57threshold, 567tiling, 420time-division, 237timing, 187training data, 529transatlantic, 173transfer matrix method, 407transition, 251transition probability, 147, 356, 607translation-invariant, 409travelling salesman problem, 246, 517tree, 242, 336, 343, 351trellis, 251, 326, 574, 577, 580, 583,608section, 251, 257termination, 579triangle, 307truth function, 211, 600tube, 257turbo code, 186, 556turbo product code, 571Turing, Alan, 265twenty questions, 70, 103twin, 111twos, 156typical evidence, 54, 60typical set, 80, 154, 363for compression, 80for noisy channel, 154typical-set decoder, 165, 230typicality, 78, 80, 162umbrella sampling, 379unbiased estimator, 307, 321, 449uncompression, 231union, 66union bound, 166, 216, 230uniquely decodeable, 93, 94units, 264universal, 110, 120, 121, 135, 590universality, in physics, 400Urbanke, Rüdiger, 570, 595urn, 31user interfaces, 118utility, 451vaccination, 458Vapnik–Chervonenkis dimension, 489variable-length code, 249, 255variable-rate error-correcting codes,238, 590variance, 1, 27, 88, 321variance–covariance matrix, 176variances add, 1, 181variational Bayes, 429variational free energy, 422, 423variational methods, 422, 433, 496,508typical properties, 435variational Gaussian process, 547VC dimension, 489vector quantization, 284, 290very good, see error-correcting codeVIBES, 431Virtakallio, Juhani, 209vision, 554visualization, 551Viterbi algorithm, 245, 329, 340, 578volume, 42, 90Von Mises distribution, 315Wainwright, Martin, 340waiting for a bus, 39, 46warning, see caution and sermonWatson–Crick base pairing, 280weather collator, 236weighing babies, 164weighing problem, 66, 68weightimportance sampling, 362in neural net, 471of binary vector, 20weight decay, 479, 529weight enumerator, 206, 211, 214, 216typical, 572weight space, 473, 474, 487Wenglish, 72, 260what number comes next?, 344white, 355white noise, 177, 179Wiberg, Niclas, 187widget, 309Wiener process, 535Wiener, Norbert, 548wife-beater, 58, 61Wilson, David B., 413, 418window, 307Winfree, Erik, 520wodge, 309Wolf, Jack, 262word-English, 260world record, 446worst-case-ism, 207, 213writing, 118Yedidia, Jonathan, 340Z channel, 148, 149–151, 155Zipf plot, 262, 263, 317Zipf’s law, 40, 262, 263Zipf, George K., 262