Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.604 B — Some Physicsor more order-parameters; in a continuous transition, all order-parameterschange continuously. [What’s an order-parameter? – a scalar function of thestate of the system; or, to be precise, the expectation of such a function.]In the vicinity of a critical point, the concept of ‘typicality’ defined inChapter 4 does not hold. For example, our toy system, at its critical point,has a 50% chance of being in a state with energy −Nɛ, and roughly a 1/2 N+1chance of being in each of the other states that have energy zero. It is thus notthe case that ln 1/P (x) is very likely to be close to the entropy of the systemat this point, unlike a system with N i.i.d. components.Remember that information content (ln 1/P (x)) and energy are very closelyrelated. If typicality holds, then the system’s energy has negligible fluctuations,and vice versa.