Information Theory, Inference, and Learning ... - MAELabs UCSD

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
9.8: Further exercises 155
given-typical-x set, 2 NH(Y |X) . So the number of non-confusable inputs, if they
are selected from the set of typical inputs x ∼ X N , is ≤ 2 NH(Y )−NH(Y |X) =
2 NI(X;Y ) .
The maximum value of this bound is achieved if X is the ensemble that
maximizes I(X; Y ), in which case the number of non-confusable inputs is
≤ 2 NC . Thus asymptotically up to C bits per cycle, and no more, can be
communicated with vanishing error probability. ✷
This sketch has not rigorously proved that reliable communication really
is possible – that’s our task for the next chapter.
9.8 Further exercises
Exercise 9.15. [3, p.159] Refer back to the computation of the capacity of the Z
channel with f = 0.15.
(a) Why is p ∗ 1
less than 0.5? One could argue that it is good to favour
the 0 input, since it is transmitted without error – and also argue
that it is good to favour the 1 input, since it often gives rise to the
highly prized 1 output, which allows certain identification of the
input! Try to make a convincing argument.
(b) In the case of general f, show that the optimal input distribution
is
p ∗ 1/(1 − f)
1 =
.
(H2(f)/(1−f))
1 + 2
(9.19)
(c) What happens to p ∗ 1
if the noise level f is very close to 1?
Exercise 9.16. [2, p.159] Sketch graphs of the capacity of the Z channel, the
binary symmetric channel and the binary erasure channel as a function
of f.
⊲ Exercise 9.17. [2 ] What is the capacity of the five-input, ten-output channel
whose transition probability matrix is
⎡
⎢
⎣
0.25
0.25
0.25
0.25
0
0
0
0
0
0
0
0.25
0.25
0.25
0.25
0
0
0
0
0
0
0
0.25
0.25
0.25
0.25
0
0
0
0
0
0
0
0.25
0.25
0.25
⎤
0.25
0.25 ⎥
0 ⎥
0 ⎥
0 ⎥
0 ⎥
0 ⎥
0 ⎥
0.25 ⎦
0
1
2
3
4
5
6
7
8
9
0 1 2 3 4
? (9.20)
0 0 0 0.25 0.25
Exercise 9.18. [2, p.159] Consider a Gaussian channel with binary input x ∈
{−1, +1} and real output alphabet AY , with transition probability density
(y−xα)2
1
Q(y | x, α, σ) = √ e− 2σ
2πσ2 2 , (9.21)
where α is the signal amplitude.
(a) Compute the posterior probability of x given y, assuming that the
two inputs are equiprobable. Put your answer in the form
P (x = 1 | y, α, σ) =
1
. (9.22)
1 + e−a(y)