Notes on computational linguistics.pdf - UCLA Department of ...

Stabler - Lx 185/209 2003
(109) If the outcomes of the binary decisions are not equally likely, though, we want to say something else.
The amount of information (or “self-information” or the “surprisal”) of an event A,
i(A) = log
1
=−log P(A)
P(A)
So if we have 10 possible events with equal probabilities of occurrence, so P(A) = 0.1, then
i(A) = log 1
=−log 0.1 ≈ 3.32
0.1
(110) The simple cases still work out properly.
In the easiest case where probability is distributed uniformly across 8 possibilities in ΩX, wewouldhave
exactly 3 bits of information given by the occurrence of a particular event A:
i(A) = log
1
=−log 0.125 = 3
0.125
The information given by the occurrence of ∪ΩX, whereP(∪ΩX) = 1, is zero:
i(A) = log 1
=−log 1 = 0
1
And obviously, if events A, B ∈ ΩX are independent, that is, P(AB) = P(A)P(B),then
i(AB) = log
1
P(AB)
1
= log P(A)P(B)
= log 1
1
P(A) + log P(B)
= i(A) + i(B)
(111) However, in the case where ΩX ={A, B} where P(A) = 0.1 andP(B) = 0.9, we will still have
i(A) = log 1
=−log 0.1 ≈ 3.32
0.1
That is, this event conveys more than 3 bits of information even though there is only one other option.
The information conveyed by the other event
i(B) = log 1
≈ .15
0.9
155