Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.Index 627poisoned glass, 103secretary, 454southeast, 520transatlantic cable, 173weighing 12 balls, 68quantum error-correction, 572queue, 454QWERTY, 119R 3, see repetition coderace, 354radial basis function, 535, 536radio, 186radix, 104RAID, 188, 190, 219, 593random, 26, 357random cluster model, 418random code, 156, 161, 164, 165, 184,192, 195, 214, 565for compression, 231random number generator, 578random variable, 26, 463random walk, 367suppression, 370, 387, 390, 395random-coding exponent, 171random-walk Metropolis method, 388rant, see sermonraptor codes, 594rate, 152rate-distortion theory, 167rateless code, 590reading aloud, 529receiver operating characteristic, 533recognition, 204record breaking, 446rectangular code, 184reducible, 373redundancy, 4, 33in channel code, 146redundant array of independent disks,188, 190, 219, 593redundant constraints in code, 20Reed–Solomon code, 185, 571, 589regression, 342, 536regret, 455regular, 557regularization, 529, 550regularization constant, 309, 479reinforcement learning, 453rejection, 364, 366, 533rejection sampling, 364adaptive, 370relative entropy, 34, 98, 102, 142, 422,429, 435, 475reliability function, 171repeat–accumulate code, 582repetition code, 5, 13, 15, 16, 46, 183responsibility, 289retransmission, 589reverse, 110reversible, 374reversible jump, 379Richardson, Thomas J., 570, 595Rissanen, Jorma, 111ROC, 533rolling die, 38roman, 127rule of thumb, 380runlength, 256runlength-limited channel, 249saccades, 554saddle-point approximation, 341sailor, 307sample, 312, 356from Gaussian, 312sampler density, 362sampling distribution, 459sampling theory, 38, 320criticisms, 32, 64sandwiching method, 419satellite communications, 186, 594scaling, 203Schönberg, 203Schottky anomaly, 404scientists, 309secret, 200secretary problem, 454security, 199, 201seek time, 593Sejnowski, Terry J., 522self-delimiting, 132self-dual, 218self-orthogonal, 218self-punctuating, 92separation, 242, 246sequence, 344sequential decoding, 581sequential probability ratio test, 464sermon, see cautionclassical statistics, 64confidence level, 465dimensions, 180gradient descent, 441illegal integral, 180importance sampling, 382interleaving, 189MAP method, 283, 306maximum entropy, 308maximum likelihood, 306most probable is atypical, 283p-value, 463sampling theory, 64sphere-packing, 209, 212stopping rule, 463turbo codes, 581unbiased estimator, 307worst-case-ism, 207set, 66shannon (unit), 265Shannon, Claude, 3, 14, 15, 152, 164,212, 215, 262, seenoisy-channel codingtheorem, source codingtheorem, informationcontentshattering, 485shifter ensemble, 524Shokrollahi, M. Amin, 568shortening, 222Siegel, Paul, 262sigmoid, 473, 527signal-to-noise ratio, 177, 178, 223significance level, 51, 64, 457, 463simplex, 173, 316Simpson’s paradox, 355Simpson, O.J., see wife-beaterssimulated annealing, 379, 392, seeannealingsix, waiting for, 38Skilling, John, 392sleep, 524, 554Slepian–Wolf, see dependent sourcesslice sampling, 374multi-dimensional, 378soft K-means clustering, 289softmax, softmin, 289, 316, 339software, xiarithmetic coding, 121BUGS, 371Dasher, 119free, xiiGaussian processes, 534hash function, 200VIBES, 431solar system, 346soldier, 241soliton distribution, 592sound, 187source code, 73, see compression,symbol code, arithmeticcoding, Lempel–Zivalgorithms, 119, 121block code, 76block-sorting compression, 121Burrows–Wheeler transform, 121for complex sources, 353for constrained channel, 249, 255for integers, 132Huffman, see Huffman codeimplicit probabilities, 102optimal lengths, 97, 102prefix code, 95software, 121stream codes, 110–130supermarket, 96, 104, 112symbol code, 91uniquely decodeable, 94variable symbol durations, 125,256source coding theorem, 78, 91, 229,231southeast puzzle, 520span, 331sparse-graph code, 338, 556density evolution, 566profile, 569sparsifier, 255species, 269spell, 201sphere packing, 182, 205sphere-packing exponent, 172Spielman, Daniel A., 568spin system, 400spines, 525spline, 538