Information Theory, Inference, and Learning ... - MAELabs UCSD

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
150 9 — Communication over a Noisy Channel
What is H(Y |X)? It is defined to be the weighted sum over x of H(Y | x);
but H(Y | x) is the same for each value of x: H(Y | x = 0) is H2(0.15),
and H(Y | x = 1) is H2(0.15). So
I(X; Y ) = H(Y ) − H(Y |X)
= H2(0.22) − H2(0.15)
= 0.76 − 0.61 = 0.15 bits. (9.8)
This may be contrasted with the entropy of the source H(X) =
H2(0.1) = 0.47 bits.
Note: here we have used the binary entropy function H2(p) ≡ H(p, 1−
p) = p log 1
p + (1 − p) log 1
(1−p)
. Throughout this book, log means
log 2 .
Example 9.6. And now the Z channel, with PX as above. P (y = 1) = 0.085.
I(X; Y ) = H(Y ) − H(Y |X)
= H2(0.085) − [0.9H2(0) + 0.1H2(0.15)]
= 0.42 − (0.1 × 0.61) = 0.36 bits. (9.9)
The entropy of the source, as above, is H(X) = 0.47 bits. Notice that
the mutual information I(X; Y ) for the Z channel is bigger than the
mutual information for the binary symmetric channel with the same f.
The Z channel is a more reliable channel.
Exercise 9.7. [1, p.157] Compute the mutual information between X and Y for
the binary symmetric channel with f = 0.15 when the input distribution
is PX = {p0 = 0.5, p1 = 0.5}.
Exercise 9.8. [2, p.157] Compute the mutual information between X and Y for
the Z channel with f = 0.15 when the input distribution is PX :
{p0 = 0.5, p1 = 0.5}.
Maximizing the mutual information
We have observed in the above examples that the mutual information between
the input and the output depends on the chosen input ensemble.
Let us assume that we wish to maximize the mutual information conveyed
by the channel by choosing the best possible input ensemble. We define the
capacity of the channel to be its maximum mutual information.
The capacity of a channel Q is:
C(Q) = max
PX
I(X; Y ). (9.10)
The distribution PX that achieves the maximum is called the optimal
input distribution, denoted by P ∗ X . [There may be multiple optimal
input distributions achieving the same value of I(X; Y ).]
In Chapter 10 we will show that the capacity does indeed measure the maximum
amount of error-free information that can be transmitted over the channel
per unit time.
Example 9.9. Consider the binary symmetric channel with f = 0.15. Above,
we considered PX = {p0 = 0.9, p1 = 0.1}, and found I(X; Y ) = 0.15 bits.