Notes on computational linguistics.pdf - UCLA Department of ...

Stabler - Lx 185/209 2003
8.1.15 Information and entropy
(108) Suppose |ΩX| =10, where these events are equally likely and partition Ω.
If we find out that X = a, how much information have we gotten?
9 possibilities are ruled out.
The possibilities are reduced by a factor of 10.
But Shannon (1948, p32) suggests that a more natural measure of the amount of information is the
number of “bits.” (A name from J.W. Tukey? Is it an acronym for BInary digiT?)
How many binary decisions would it take to pick one element out of the 10? We can pick 1 out of 8
with 3 bits; 1 out of 16 with 4 bits; so 1 out of 10 with 4 (and a little redundancy). More precisely, the
number of bits we need is log2(10) ≈ 3.32.
Exponentiation and logarithms review
km · kn = km+n k0 = 1
k−n = 1
kn am an = am−n logk x = y iff ky = x
logk(kx ) = x since: kx = kx and so: logk k = 1
and: logk 1 = 0
logk( M
N ) = logk M − logk N
logk(MN) = logk M + logk N
so, in general: logk(M p ) = p · logk M
1
and we will use: logk x = logk x−1 =−1 · logk x =−logk x
E.g. 512 = 29 and so log2 512 = 9. And log10 3000 = 3.48 = 103 · 100.48 .And5−2 = 1
We’ll stick to log2 and “bits,” but another common choice is loge,where e = lim(1
+ x)
x→0 1
x =
∞
n=0
1
≈ 2.7182818284590452
n!
25 ,solog 5
1
25 =−2
Or, more commonly, e is defined as the x such that a unit area is found under the curve 1
from u = 1to
u
u = x, that is, it is the positive root x of x 1
1 u du = 1.
This number is irrational, as shown by the Swiss mathematician Leonhard Euler (1707-1783), in whose
honor we call it e. In general: ex = ∞ x
k=0
k
k! . And furthermore, as Euler discovered, eπ√ −1 + 1 = 0. This is
sometimes called the most famous of all formulas (Maor, 1994). It’s not, but it’s amazing.
Using log 2 gives us “bits,” log e givesus“nats,”andlog 10 givesus“hartleys.”
153