Information Theory, Inference, and Learning ... - Inference Group

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.17.3: Counting the number of possible messages 25111 00(a) 0 (c)s 1s 2s 3s 41❢1❢1❢1❢1❢1❢1❢1❢0 0 0 0 0 0 00❢ ✒1 ❅❅❅❘✲ ❢ ✒1 ❅❅❅❘00 ✲ ❢ ✒1 ❅❅❅❘00 ✲ ❢ ✒1 ❅❅❅❘00 ✲ ❢ ✒1 ❅❅❅❘00 ✲ ❢ ✒1 ❅❅❅❘00 ✲ ❢ ✒1 ❅00 ✲ ❢ ✒1 ❅❅❘00 ✲ ❢ 00s 5s 6s 7s 8(b)1❥❅01❥❅❅❅❘0❥ ✒✲ ❥0 0(d) A =s n+1s n1111 0101 00 00B A =s n s n+111♠1 ❆ 1✲ 11♠11❆❆❆❆❆❆❆❯01♠ ✒ 11♠11 00♠1 ♠ 100❅00❅❅❅❘✁ ✁✁✁✁✁✁✁✕ 0000♠✲ ♠0 00⎡⎤1 1 0 0⎢ 0 0 0 1⎣ 1 0 0 00 0 1 10(to)10⎥⎦ C A =(from)[ 1 0 ] 0 11 1s n+1s n11♥❆ 1 11 ♥❆❆❆❆❆❆❆❯01♥ ✒❅01 1♥❅❅❅❘0♥ ✒10❅0♥❅❅❅❘✁ ✁✁✁✁✁✁✁✕00♥00 ♥⎡⎢⎣0 1 0 00 0 1 11 1 0 00 0 1 0⎤⎥⎦Figure 17.2. (a) State diagram forchannel A. (b) Trellis section. (c)Trellis. (d) Connection matrix.Figure 17.3. State diagrams, trellissections and connection matricesfor channels B and C.so in this chapter we will denote the number of distinguishable messages oflength N by M N , and define the capacity to be:1C = limN→∞ N log M N. (17.10)Once we have figured out the capacity of a channel we will return to thetask of making a practical code for that channel.17.3 Counting the number of possible messagesFirst let us introduce some representations of constrained channels. In a statediagram, states of the transmitter are represented by circles labelled with thename of the state. Directed edges from one state to another indicate thatthe transmitter is permitted to move from the first state to the second, and alabel on that edge indicates the symbol emitted when that transition is made.Figure 17.2a shows the state diagram for channel A. It has two states, 0 and1. When transitions to state 0 are made, a 0 is transmitted; when transitionsto state 1 are made, a 1 is transmitted; transitions from state 1 to state 1 arenot possible.We can also represent the state diagram by a trellis section, which showstwo successive states in time at two successive horizontal locations (figure17.2b). The state of the transmitter at time n is called s n . The set ofpossible state sequences can be represented by a trellis as shown in figure 17.2c.A valid sequence corresponds to a path through the trellis, and the number of