Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.626 Indexneuron, 471capacity, 483Newton, Isaac, 200, 552Newton–Raphson method, 303, 441nines, 198noise, 3, see channelcoloured, 179spectral density, 177white, 177, 179noisy channel, see channelnoisy typewriter, 148, 152, 154noisy-channel coding theorem, 15,152, 162, 171, 229Gaussian channel, 181linear codes, 229poor man’s version, 216noisy-or, 294non-confusable inputs, 152noninformative, 319nonlinear, 535nonlinear code, 20, 187nonparametric data modelling, 538nonrecursive, 575noodle, Buffon’s, 38normal, 312, see Gaussiannormal graph, 219, 584normalizing constant, see partitionfunctionnot-sum, 335notation, 598absolute value, 33, 599conventions of this book, 147convex/concave, 35entropy, 33error function, 156expectation, 37intervals, 90logarithms, 2matrices, 147probability, 22, 30set size, 33, 599transition probability, 147vectors, 147NP-complete, 184, 325, 517nucleotide, 201, 204nuisance parameters, 319numerology, 208Nyquist sampling theorem, 178objective function, 473Occam factor, 322, 345, 348, 350, 352Occam’s razor, 343octal, 575octave, 478odds, 456Ode to Joy, 203officer for whimsical departmentalrules, 464Oliver, 56one-way hash function, 200optic nerve, 491optimal decoder, 152optimal input distribution, 150, 162optimal linear filter, 549optimal stopping, 454optimization, 169, 392, 429, 479, 505,516, 531gradient descent, 476Newton algorithm, 441of model complexity, 531order parameter, 604ordered overrelaxation, 391orthodox statistics, 320, see samplingtheoryouter code, 184overfitting, 306, 322, 529, 531overrelaxation, 390p-value, 64, 457, 462packet, 188, 589paradox, 107Allais, 454bus-stop, 39heat capacity, 401Simpson’s, 355waiting for a six, 38paranormal, 233parasite, 278parent, 559parity, 9parity-check bits, 9, 199, 203parity-check code, 220parity-check constraints, 20parity-check matrix, 12, 183, 229, 332generalized, 581parity-check nodes, 19, 219, 567, 568,583parse, 119, 448Parsons code, 204parthenogenesis, 273partial order, 418partial partition functions, 407particle filter, 396partition, 174partition function, 401, 407, 409, 422,423, 601, 603analogy with lake, 360partial, 407partitioned inverse, 543path-counting, 244pattern recognition, 156, 179, 201pentagonful code, 21, 221perfect code, 208, 210, 211, 219, 589perfect simulation, 413periodic variable, 315permutation, 19, 268Petersen graph, 221phase transition, 361, 403, 601philosophy, 26, 119, 384phone, 125, 594cellular, see mobile phonephone directory, 193phone number, 58, 129photon counter, 307, 342, 448physics, 80, 85, 257, 357, 401, 422,514, 601pigeon-hole principle, 86, 573pitchfork bifurcation, 291, 426plaintext, 265plankton, 359point estimate, 432point spread function, 549pointer, 119poisoned glass, 103Poisson distribution, 2, 175, 307, 311,342Poisson process, 39, 46, 448Poissonville, 39, 313polymer, 257poor man’s coding theorem, 216porridge, 280portfolio, 455positive definite, 539positivity, 551posterior probability, 6, 152power cost, 180power law, 584practical, 183, see error-correctingcodeprecision, 176, 181, 312, 320, 383precisions add, 181prediction, 29, 52predictive distribution, 111prefix code, 92, 95prior, 6, 308, 529assigning, 308improper, 353Jeffreys, 316subjectivity, 30prior equivalence, 447priority of bits in a message, 239prize, on game show, 57probabilistic model, 111, 120probabilistic movie, 551probability, 26, 38Bayesian, 50contrasted with likelihood, 28density, 30, 33probability distributions, 311, seedistributionprobability of block error, 152probability propagation, seesum–product algorithmproduct code, 184, 214profile, of random graph, 568pronunciation, 34proper, 539proposal density, 364, 365Propp, Jim G., 413, 418prosecutor’s fallacy, 25prospecting, 451protein, 204, 269regulatory, 201, 204synthesis, 280protocol, 589pseudoinverse, 550Punch, 448puncturing, 222, 580pupil, 553puzzle, see gamecable labelling, 173chessboard, 520fidelity of DNA replication, 280hat, 222, 223life, 520magic trick, 233, 234