1 Introduction

1.6. Information Theory 51the number of different ways of allocating the objects to the bins. There are Nways to choose the first object, (N − 1) ways to choose the second object, andso on, leading to a total of N! ways to allocate all N objects to the bins, where N!(pronounced ‘factorial N’) denotes the product N ×(N −1)×···×2×1. However,we don’t wish to distinguish between rearrangements of objects within each bin. Inthe i th bin there are n i ! ways of reordering the objects, and so the total number ofways of allocating the N objects to the bins is given byW = ∏N!i n (1.94)i!which is called the multiplicity. The entropy is then defined as the logarithm of themultiplicity scaled by an appropriate constantH= 1 N ln W = 1 N ln N! − 1 ∑ln n i !. (1.95)NWe now consider the limit N →∞, in which the fractions n i /N are held fixed, andapply Stirling’s approximationwhich givesH=− limN→∞iln N! ≃ N ln N − N (1.96)∑i( ni) ( ni) ∑ln = −N Nip i ln p i (1.97)where we have used ∑ i n i = N. Here p i = lim N→∞ (n i /N ) is the probabilityof an object being assigned to the i th bin. In physics terminology, the specific arrangementsof objects in the bins is called a microstate, and the overall distributionof occupation numbers, expressed through the ratios n i /N , is called a macrostate.The multiplicity W is also known as the weight of the macrostate.We can interpret the bins as the states x i of a discrete random variable X, wherep(X = x i )=p i . The entropy of the random variable X is thenH[p] =− ∑ ip(x i )lnp(x i ). (1.98)Appendix EDistributions p(x i ) that are sharply peaked around a few values will have a relativelylow entropy, whereas those that are spread more evenly across many values willhave higher entropy, as illustrated in Figure 1.30. Because 0 p i 1, the entropyis nonnegative, and it will equal its minimum value of 0 when one of the p i =1 and all other p j≠i =0. The maximum entropy configuration can be found bymaximizing H using a Lagrange multiplier to enforce the normalization constrainton the probabilities. Thus we maximize˜H ∑ ( )∑=− p(x i )lnp(x i )+λ p(x i ) − 1(1.99)ii