B. P. Lathi, Zhi Ding - Modern Digital and Analog Communication Systems-Oxford University Press (2009)

748 INTRODUCTION TO INFORMATION THEORY
simply by increasing T. In the limit as T ➔ oo, the occupancy factor approaches 0. This will
make the error probability go to 0, and we have the possibility of error-free communication.
One important question, however, still remains unanswered. What must be the rate reduction
ratio a/ fJ for this dream to come true? To answer this question, we observe that increasing
T increases the length of the transmitted sequence (fJT digits). If P e is the digit error probability,
then it can be seen from the relative frequency definition ( or the law of large numbers)
that as T ➔ oo, the total number of digits in error in a sequence of (JT digits (fJT ➔ oo) is
exactly fJTP e . Hence, the received sequences will be at a Hamming distance of fJTP e from the
transmitted sequences. Therefore, for error-free communication, we must leave all the vertices
unoccupied within spheres of radius fJTP e drawn around each of the 2 aT occupied vertices.
In short, we must be able to pack 2 aT nonoverlapping spheres, each of radius fJ TP e, into the
Hamming space of dimensions fJT. This means that for a given (J, a cannot be increased
beyond some limit without causing overlap in the spheres and the consequent failure of the
error correction scheme. Shannon's theorem states that for this scheme to work, a/ fJ must
be less than the constant (channel capacity) C s , which physically is a function of the channel
noise and the signal power:
a
- < C s (1 3.16)
fJ
It must be remembered that such perfect, error-free communication is not practical. In
this system we accumulate the information digits for T seconds before encoding them, and
because T ➔ oo, for error-free communication we would have to wait until eternity to start
encoding. Hence, there will be an infinite delay at the transmitter and an additional delay of
the same amount at the receiver. Second, the equipment needed for the storage, encoding,
and decoding sequence of infinite digits would be monstrous. Needless to say, the dream of
error-free communication cannot be achieved in practice. Then what is the use of Shannon's
result? For one thing, it indicates the upper limit on the rate of error-free communication that
can be achieved on a channel. This in itself is monumental. Second, it indicates that we can
reduce the error probability below an arbitrarily small level by allowing only a small reduction
in the rate of transmission of information digits. We can therefore seek a compromise between
error-free communication with infinite delay and virtually error-free communication with a
finite delay.
13.4 CHANNEL CAPACITY OF A DISCRETE
MEMORYLESS CHANNEL
This section treats discrete memoryless channels. Consider a source that generates a message
that contains r symbols x1, x2, ... , x,. The receiver receives symbols Yl, Y2, ... , Y s· The set
of symbols {Yk} may or may not be identical to the set {xk }, depending on the nature of the
receiver. If we use receivers of the types discussed in Chapter 10, the set of received symbols
will be the same as the set transmitted. This is because the optimum receiver, upon receiving a
signal, decides which of the r symbols x1, x2, ... , x, has been transmitted. Here we shall be
more general and shall not constrain the set {Yk} to be identical to the set {xk}.
If the channel is noiseless, then the reception of some symbol Yj uniquely determines
the message transmitted. Because of noise, however, there is a certain amount of uncertainty
regarding the transmitted symbol when Yj is received. If P(xi lYj ) represents the conditional
probabilities that Xi was transmitted when Yj is received, then there is an uncertainty of
log [ 1 / P (xi lyj )] about Xi when Yj is received. When this uncertainty is averaged over all