Notes on computational linguistics.pdf - UCLA Department of ...

Stabler - Lx 185/209 2003
(36) Theorem: P(A) = 1 − P(A)
Proof: Obviously AandAare disjoint, so by axiom iii, P(A∪ A) = P(A) + P(A)
Since A ∪ A = Ω, axiom ii tells us that P(A) + P(A) = 1
(37) Theorem: P(∅) = 0
(38) Theorem: P(A∪ B) = P(A) + P(B)− P(A∩ B)
Proof: Since A is the union of disjoint events (A ∩ B) and (B ∩ A), P(A) = P(A∩ B) + (B ∩ A).
Since B is the union of disjoint events (A ∩ B) and (A ∩ B), P(B) = P(A∩ B) + (A ∩ B).
And finally, since (A ∪ B) is the union of disjoint events (B ∩ A), (A ∩ B) and (A ∩ B), P(A ∪ B) =
P(B ∩ A) + P(A∩ B) + P(A ∩ B).
Now we can calculate P(A) + P(B) = P(A ∩ B) + (B ∩ A) + P(A ∩ B) + (A ∩ B), andsoP(A ∪ B) =
P(A)+ P(B)− P(A∩ B).
(39) Theorem (Boole’s inequality): P( ∞ 0 Ai) ≤ ∞ 0 P(Ai)
(40) Exercises
a. Prove that if A ⊆ B then P(A) ≤ P(B)
b. In (38), we see what P(A∪ B) is. What is P(A∪ B ∪ C)?
c. Prove Boole’s inequality.
(41) The conditional probability of A given B, P(A|B) =df P(A∩B)
P(B)
(42) Bayes’ theorem: P(A|B) = P(A)P(B|A)
P(B)
Proof: From the definition of conditional probability just stated in (41), (i) P(A∩B) = P(B)P(A|B). The
definition of conditional probability (41) also tells us P(B|A) = P(A∩B)
P(A) , and so (ii) P(A∩B) = P(A)P(B|A).
Given (i) and (ii), we know P(A)P(B|A) = P(B)P(A|B), from which the theorem follows immediately.
The English mathematician Thomas Bayes (1702-1761) was a Presbyterian minister. He distributed some papers,
and published one anonymously, but his influential work on probability, containing a version of the theorem
above, was not published until after his death.
Bayes is also associated with the idea that probabilities may be regarded as degrees of belief, and this has inspired
recent work in models of scientific reasoning. See, e.g. Horwich (1982), Earman (1992), Pearl (1988).
In fact, in a Microsoft advertisement we are told that their Lumiere Project uses “a Bayesian perspective on integrating
information from user background, user actions, and program state, along with a Bayesian analysis of
the words in a user’s query…this Bayesian information-retrieval component of Lumiere was shipped in all of the
Office ’95 products as the Office Answer Wizard…As a user works, a probability distribution is generated over
areas that the user may need assistance with. A probability that the user would not mind being bothered with
assistance is also computed.”
See, e.g. http://www.research.microsoft.com/research/dtg/horvitz/lum.htm.
For entertainment, and more evidence of the Bayesian cult that is sweeping certain subcultures, see, e.g. http://www.afit.af.m
For some more serious remarks on Microsoft’s “Bayesian” spelling correction, and a new proposal inspired by
trigram and Bayesian methods, see e.g. Golding and Schabes (1996).
For some serious proposals about Bayesian methods in perception: Knill and Richards (1996); and in language
acquisition: Brent and Cartwright (1996), de Marcken (1996).
(43) A and B are independent iff P(A∩ B) = P(A)P(B).
133