Notes on computational linguistics.pdf - UCLA Department of ...

Stabler - Lx 185/209 2003
Entropy
(112) Often we are interested not in the information conveyed by a particular event, but by the information
conveyed by an information source:
…from the point of view of engineering, a communication system must face the problem of
handling any message that the source can produce. If it is not possible or practicable to design
a system which can handle everything perfectly, then the system should handle well the jobs
it is most likely to be asked to do, and should resign itself to be less efficient for the rare task.
This sort of consideration leads at once to the necessity of characterizing the statistical nature
of the whole ensemble of messages which a given kind of source can and will produce. And
information, as used in communication theory, does just this. (Weaver, 1949, p14)
(113) For a source X, the average information of an arbitrary outcome in ΩX is
H =
P(A)i(A) =−
P(A)log P(A)
A∈ΩX
(114)
This is sometimes called “entropy” of the random variable – the average number of bits per event
(Charniak, 1993, p29). So called because each P(A) gives us the “proportion” of times that A occurs.
For a source X of an infinite sequence of events, the entropy or average information, the entropy of the
source is usually given as their average probability over an infinite sequence X1,X2,..., easily calculated
from the previous formula to be:
Gn
H(X) = lim
n→∞ n
where Gn =
−n
...
P(X1 = A1,X2 = A2,...,Xn = An) log P(X1 = A1,X2 = A2,...,Xn = An)
A1∈ΩX A2∈ΩX
An∈ΩX
(115) When the space ΩX consists of independent time-invariant events whose union has probability 1, then
Gn =−n
P(A)log P(A),
A∈ΩX
A∈ΩX
and so the entropy or average information of the source in the following way:
H(X) =
P(A)i(A) =−
P(A)log P(A)
A∈ΩX
Charniak (1993, p29) calls this the per word entropy of the process.
(116) If we use some measure other than bits, a measure that allows r -ary decisions rather than just binary
ones, then we can define Hr (X) similarly except that we use logr rather than log2. (117) Shannon shows that this measure of information has the following intuitive properties (as discussed
also in the review of this result in Miller and Chomsky (1963, pp432ff)):
a. Adding any number of impossible events to ΩX does not change H(X).
b. H(X) is a maximum when all the events in ΩX are equiprobable.
(see the last graph on page 154)
c. H(X) is additive, in the sense that H(Xi ∪ Xj) = H(Xi) + H(Xj) when Xi and Xj are independent.
156
A∈ΩX