Information Theory, Inference, and Learning ... - MAELabs UCSD

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
258 17 — Communication over Constrained Noiseless Channels
Solution to exercise 17.10 (p.255). Let the invariant distribution be
P (s) = αe (L)
s e(R)
s , (17.22)
where α is a normalization constant. The entropy of St given St−1, assuming Here, as in Chapter 4, St denotes
St−1 comes from the invariant distribution, is
H(St|St−1) = −
the ensemble whose random
variable is the state st.
= −
s,s ′
α e (R)
s
s,s ′
= −
αe (L)
s,s ′
e (L)
s ′ As ′
s
log e
λ
(L)
s ′
P (s)P (s ′ |s) log P (s ′ |s) (17.23)
s e (R)
s
e (L)
s ′ As ′ s
λe (L)
s
log e(L)
s ′ As ′ s
λe (L)
s
(17.24)
+ log As ′ s − log λ − log e (L)
s . (17.25)
Now, As ′ s is either 0 or 1, so the contributions from the terms proportional to
As ′ s log As ′ s are all zero. So
H(St|St−1) = log λ + − α
λ
s ′
s
α
e
λ
(L)
s ′ As ′
s
= log λ − α
λ
s ′
s
s ′
λe (R)
s ′ e (L)
s ′ log e (L)
s ′ + α
λ
As ′ se (R)
s
e (R)
s log e (L)
s
s
e (L)
s ′ log e (L)
s ′ +
(17.26)
λe (L)
s e(R) s log e (L)
s (17.27)
= log λ. (17.28)
Solution to exercise 17.11 (p.255). The principal eigenvalues of the connection
matrices of the two channels are 1.839 and 1.928. The capacities (log λ) are
0.879 and 0.947 bits.
Solution to exercise 17.12 (p.256). The channel is similar to the unconstrained
binary channel; runs of length greater than L are rare if L is large, so we only
expect weak differences from this channel; these differences will show up in
contexts where the run length is close to L. The capacity of the channel is
very close to one bit.
A lower bound on the capacity is obtained by considering the simple
variable-length code for this channel which replaces occurrences of the maximum
runlength string 111. . .1 by 111. . .10, and otherwise leaves the source file
unchanged. The average rate of this code is 1/(1 + 2 −L ) because the invariant
distribution will hit the ‘add an extra zero’ state a fraction 2 −L of the time.
We can reuse the solution for the variable-length channel in exercise 6.18
(p.125). The capacity is the value of β such that the equation
L+1
Z(β) = 2 −βl = 1 (17.29)
l=1
is satisfied. The L+1 terms in the sum correspond to the L+1 possible strings
that can be emitted, 0, 10, 110, . . . , 11. . .10. The sum is exactly given by:
Z(β) = 2 −β
2−β L+1 − 1
2−β . (17.30)
− 1