B. P. Lathi, Zhi Ding - Modern Digital and Analog Communication Systems-Oxford University Press (2009)

798 INTRODUCTION TO INFORMATION THEORY
TABLE P. 13. 1-5
Probability of Occurrence of Letters in the English
Lang uage
letter
Space
E
T
A
0
N
R
I
s
H
D
L
F
C
M
u
G
y
p
w
B
V
K
X
J
Q
z
Probability - log P;
0.187 2.46
0.1073 3.22
0.0856 3.84
0.0668 3.90
0.0654 3.94
0.0581 4.11
0.0559 4.16
0.0519 4.27
0.0499 4.33
0.04305 4.54
0.03 100 5.02
0.02775 5.17
0.02395 5.38
0.02260 5.45
0.02075 5.60
0.02010 5.64
0.01633 5.94
0.01623 5.95
0.01623 5.95
0.01620 6.32
0.01179 6.42
0.00752 7.06
0.00344 8.20
0.00136 9.54
0.00108 9.85
0.00099 9.98
0.00063 10.63
(c) Use Zipf's law relating the word rank to its probability. In English prose, if we order words
according to the frequency of usage so that the most frequently used word (the) is word
number 1 (rank 1), the next most probable word (o/) is number 2 (rank 2), and so on, then
empirically it is found that P(r), the probability of the rth word (rank r) is very nearly
0.1
P(r) = - r
Now use Zipf's law to compute the entropy per word. Assume that there are 8727 words.
The reason for this number is that the probabilities P(r) sum to 1 for r from 1 to 8727. Zipf's
law, surprisingly, gives reasonably good results. Assuming there are 5.5 letters (including
space) per word on the average, determine the entropy or information per letter.
13.2-1 A source emits seven messages with probabilities 1/2, 1/4, 1/8, 1/16, 1/32, 1/64, and 1/64,
respectively. Find the entropy of the source. Obtain the compact binary code and find the average
length of the codeword. Determine the efficiency and the redundancy of the code.
13.2-2 A source emits seven messages with probabilities 1/3, 1/3, 1/9, 1/9, 1/27, 1/27, and 1/27, respectively.
Find the entropy of the source. Obtain the compact 3-ary code and find the average length
of the codeword. Determine the efficiency and the redundancy of the code.