A Course on Large Deviations with an Introduction to Gibbs Measures.

6 1. Introduction
sequence with [hn] zeros and ones, then we have 2 [hn] possible words to use.
One should then have
n
≤ 2
[ns]
[hn] .
Using Stirling’s formula (or directly applying (1.1)) one obtains that the
minimal possible h is
with I 1/2 given in (1.1).
h(s) = −s log 2 s − (1 − s) log 2(1 − s) = 1 − I 1/2(s)
log 2 ,
Note that h(0) = h(1) = 0. This makes sense, since if we know s = 0
(respectively, s = 1) then we know we will only get zeros (respectively, ones).
Thus, there is nothing to encode. This is the case of complete order. On the
other hand, h(1/2) = 1. This too makes sense since there is no information
one can extract from a sequence of fair coin tosses and one needs all n
bits to encode the sequence. This is the case of complete disorder. For
s ∈ (0, 1/2), one knows that a 1 is less likely to occur than a 0 and hence
one should be able to conserve and use fewer than n bits to encode the
sequence. However, the above formula says that one cannot do better than
h(s)n. This, of course, does not tell us what the best encoding algorithm is.
1.2. Thermodynamic entropy
Once again, for the sake of illustration, we will describe an oversimplified
system. This section is inspired by Schrödinger’s course [34].
Consider a physical system of n independent identical components. By
independent we mean that the components do not communicate with each
other. By identical we mean that each of them has the same “mechanism”
attached to it, screws, pistons, and what not. Each component can be at an
energy level from the set {εℓ : ℓ ∈ N}. We submit the system to a heat bath
at a fixed absolute temperature T which causes it to have total energy E. Let
aℓ be the number of components in state εℓ. The system tries to maximize
its disorder by choosing aℓ’s so that the number of possible configurations
n!
a1!···aℓ!··· is as large as possible, subject to the constraints aℓ = n and
aℓεℓ = E.
Equivalently, one can maximize the logarithm of the quantity in question
and use Lagrange multipliers to achieve this optimization task; see page 266
of Bartle’s textbook [3]. First, one sets the gradient of
log
n!
a1! · · · aℓ! · · · − α aℓ − β aℓεℓ