Notes on computational linguistics.pdf - UCLA Department of ...

Stabler - Lx 185/209 2003
With this code we need only about 2.58 bits per symbol
(126) Consider
12123333123333123312
Here we have P(1) = P(2) = 1
1
4 and P(3) = 2 ,sotheentropyis1.5/bitspersymbol.
The sequence has length 20, so we should be able to encode it with 30 bits.
However, consider blocks of 2. P(12) = 1
1
, P(33) = , and the entropy is 1 bit/symbol.
2 2
For the sequence of 10 blocks of 2, we need only 10 bits.
So it is often worth looking for structure in larger and larger blocks.
8.1.17 Kraft’s inequality and Shannon’s theorem
(127) MacMillan:
(128) Kraft (1949):
If uniquely decodable code C has K codewords of lengths l1,...,lK then
K
2 −li ≤ 1.
i=1
If a sequence l1,...,lK satisfies the previous inequality, then there is a uniquely decodable
code C that has K codewords of lengths l1,...,lK
(129) Shannon’s theorem. Using the definition of Hr in (116), Shannon (1948) proves the following famous
theorem which specifies the information-theoretic limits of data compression:
Suppose that X is a first order source with outcomes (or outputs) ΩX. Encoding the characters
of ΩX in a code with characters Γ where |Γ |=r>1 requires an average of Hr (X) characters
of Γ per character of ΩX.
Furthermore, for any real number ɛ>0, there is a code that uses an average of Hr (X) + ɛ
characters of Γ per character of ΩX.
8.1.18 String edits and other varieties of sequence comparison
Overviews of string edit distance methods are provided in Hall and Dowling (1980) and Kukich (1992).
Masek and Paterson (1980) present a fast algorithm for computing string edit distances.
Ristad (1997) and Ristad and Yianilos (1996) consider the problem of learning string edit distances.
8.2 Probabilisitic context free grammars and parsing
8.2.1 PCFGs
(130) A probabilistic context free grammar (PCFG)
where
G =〈Σ,N,(→), S,P〉,
1. Σ,N are finite, nonempty sets,
2. S is some symbol in N,
3. the binary relation (→) ⊆ N × (Σ ∪ N) ∗ is also finite (i.e. it has finitely many pairs),
159