Introduction to Acoustics

number of possibilities
H = log Ω. (14.166)
Because log 100 is just twice log 10, the logical problem
is solved. The information measured in bits is obtained
by using a base 2 logarithm.
A few simple features follow immediately. If the
number of possible messages is Ω = 1 then the message
provides no information, which agrees with log 1 = 0. If
the context is binary, where a character can be only 1 or
0(Ω = 2), then receiving a character provides 1 bit of
information, which agrees with log2 2 = 1.
If the context is an alphabet with M possible symbols,
and all of the symbols are equally probable, then
a message with N characters has Ω = M N possible
outcomes and the information entropy is
H = log M N = N log M , (14.167)
illustrating the additivity of information over the characters
of the message.
14.14.1 Shannon Entropy
Information theory becomes interesting when the probabilities
of different symbols are different. Shannon [14.3,
4] showed that the information content per character is
given by
M�
pi log pi , (14.168)
Hc =−
i=1
where pi is the probability of symbol i in the given
context.
The rest of this section proves Shannon’s formula.
The proof begins with the plausible assumption that, if
the probability of symbol i is pi, then in a very long
message of N characters, the number of occurrences of
character i, mi will be exactly mi = Npi.
The number of possibilities for a message of N
characters in which the set of {mi} is fixed by the
corresponding {pi} is
N!
Ω =
. (14.169)
m1! m2! ... m M!
Therefore,
H = log N!−log m1!−log m2!−...log m M! .
(14.170)
One can write log N! as a sum
N�
log N!= log k (14.171)
k=1
and similarly for log mi!.
Acoustic Signal Processing 14.14 Information Theory 529
For a long message one can replace the sum by an
integral,
�
log N!=
and similarly for log mi! .
Therefore,
1
i=1
N
dx log x = N log N − N + 1 (14.172)
H =N log N − N + 1
M�
M�
− mi log mi +
i=1
mi −
M�
1 . (14.173)
i=1
Because � M i=1 mi = N, this reduces to
H = N log N + 1 −
M�
mi log mi − M . (14.174)
i=1
The information per character is obtained by dividing
the message entropy by the number of characters in the
message,
Hc = log N −
M�
pi log mi + (1 − M)/N ,
i=1
(14.175)
where we have used the fact that mi/N = pi.
In a long message, the last term can be ignored as
small. Then because the sum of probabilities pi is equal
to 1,
M�
Hc =− pi(log mi − log N) , (14.176)
or
i=1
Hc =−
i=1
M�
pi log pi , (14.177)
which is (14.168) as advertised.
If the context of written English consists of 27
symbols (26 letters and a space), and if all symbols
are equally probable, then the information content of
a single character is
Hc =−1.443
27�
i=1
1 1
ln = 4.75 (bits) , (14.178)
27 27
where the factor 1/ ln(2) = 1.443 converts the natural
log to a base 2 log. However, in written English all symbols
are not equally probable. For example, the most
Part D 14.14