B. P. Lathi, Zhi Ding - Modern Digital and Analog Communication Systems-Oxford University Press (2009)

13.2 Source Encoding 739
on the average, for encoding. From the intuitive standpoint, on the other hand, information is
thought of as being synonymous with the amount of surprise, or uncertainty, associated with
the event ( or message). A smaller probability of occurrence implies more uncertainty about the
event. Uncertainty is, of course, associated with surprise. Hence intuitively, the information
associated with a message is a measure of the uncertainty (unexpectedness) of the message.
Therefore, log (1/ Pi) is a measure of the uncertainty of the message m;, and I:7= 1 P; log (1/P;)
is the average uncertainty (per message) of the source that generates messages m1, m2, ... ,
m n with probabilities P1 , P2, ... , P11 • Both these interpretations prove useful in the qualitative
understanding of the mathematical definitions and results in information theory. Entropy may
also be viewed as a function associated with a random variable m that assumes values m1,
m2, ... , m n with probabilities P(m1), P(m2), ... , P(m11):
n
I
n
1
L.,
i=l
P(m·) I L,
i=l
P· I
H(m) = " P(m;) log -- = " Pi log -
Thus, we can associate an entropy with every discrete random variable.
If the source is not memoryless (i.e., in the event that a message emitted at any time is
dependent of the previous messages emitted), then the source entropy will be less than H (m)
in Eq. (1 3.9). This is because the dependence of a message on previous messages reduces its
uncertainty.
13.2 SOURCE ENCODING
The minimum number of binary digits required to encode a message was shown to be equal to
the source entropy log(l/P) if all the messages of the source are equiprobable (each message
probability is P). We shall now generalize this result to the case of nonequiprobable messages.
We shall show that the average number of binary digits per message required for encoding is
given by H (m) (in bits) for an arbitrary probability distribution of the messages.
Let a source m emit messages m1, m2, ... , m ,, with probabilities P1, P2, ... , P n , respectively.
Consider a sequence of N messages with N ➔ oo. Let k; be the number of times
message m; occurs in this sequence. Then according to the relative frequency interpretation
( or law of large numbers),
k·
lim _!_ = P;
N ➔oo N
Thus, the message m; occurs NP; times in a sequence of N messages (provided N ➔ oo ).
Therefore, in a typical sequence of N messages, m1 will occur NP1 times, m2 will occur
NP2 times, ... , m ,, will occur NP ,, times. All other compositions are extremely unlikely to
occur (P ➔ 0). Thus, any typical sequence (where N ➔ oo) has the same proportion of the
n messages, although in general the order will be different. We shall assume a memoryless
source; that is, we assume that the message is emitted from the source independently of the
previous messages. Consider now a typical sequence SN of N messages from the source.
Because the n messages (of probability P1, P2, ... , P ,, ) occur NP1, NP2, ... , NP ,, times, and
because each message is independent, the probability of occurrence of a typical sequence SN
is given by
(13.11)