Information Theory, Inference, and Learning ... - MAELabs UCSD

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
604 B — Some Physics
or more order-parameters; in a continuous transition, all order-parameters
change continuously. [What’s an order-parameter? – a scalar function of the
state of the system; or, to be precise, the expectation of such a function.]
In the vicinity of a critical point, the concept of ‘typicality’ defined in
Chapter 4 does not hold. For example, our toy system, at its critical point,
has a 50% chance of being in a state with energy −Nɛ, and roughly a 1/2 N+1
chance of being in each of the other states that have energy zero. It is thus not
the case that ln 1/P (x) is very likely to be close to the entropy of the system
at this point, unlike a system with N i.i.d. components.
Remember that information content (ln 1/P (x)) and energy are very closely
related. If typicality holds, then the system’s energy has negligible fluctuations,
and vice versa.