Notes on computational linguistics.pdf - UCLA Department of ...

Stabler - Lx 185/209 2003
One indirect argument for stochastic models of this kind could come from the presentation of a theory
of human language acquisition based on stochastic grammars.
8.1.16 Codes
(121) Shannon considers the information in a discrete, noiseless message. Here, the space of possible events
ΩX isgivenbyanalphabet(or“vocabulary”)Σ.
A fundamental result is Shannon’s result that the entropy of the source sets a lower bound on the size
of the messages. We present this result in §129 below after setting the stage with the basic ideas we
need.
(122) Sayood (1996, p26) illustrates some basic points about codes with some examples. Consider:
message code 1 code 2 code 3 code 4
a 0 0 0 0
b 0 1 10 01
c 1 00 110 011
d 10 11 111 0111
avg length 1.125 1.125 1.75 1.875
Notice that baa in code 2 is 100. But 100 is also the encoding of bc.
We might like to avoid this. Codes 3 and 4 have the nice property of unique decodability. That is, the
map from message sequences to code sequences is 1-1.
(123) Consider encoding the sequence
91111111413151716172021
a. To transmit these numbers in binary code, we would need 5 bits per element.
b. To transmit 9 different digits: 9, 11, 13, 14, 15, 16, 17, 20, 21, we could hope for a somewhat better
code! 4 bits would be more than enough.
c. An even better idea: notice that the sequence is close to the function f(n)= n+8 forn∈{1, 2,...}
The perturbation or residual Xn−f(n)= 0, 1, 0, −1, 1, −1, 0, 1, −1, −1, 1, 1, so it suffices to transmit
the perturbation, which only requires two bits.
(124) Consider encoding the sequence,
27 28 29 28 26 27 29 28 30 32 34 36 38
This sequence does not look quite so regular as the previous case.
However, each value is near the previous one, so one strategy is to let your receiver know the starting
point and then send just the changes:
(27)11-1-212-122222
(125) Consider the follow sequence of 41 elements, generated by a probabilistic source:
axbarayaranxarrayxranxfarxfaarxfaaarxaway
Thereare8symbolshere,sowecoulduse3bitspersymbol.
On the other hand, we could use the following variable length code:
a 1
x 001
b 01100
f 0100
n 0111
r 000
w 01101
y 0101
158