Information Theory, Inference, and Learning ... - MAELabs UCSD

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
60 3 — More about Inference
8
6
4
2
0
-2
-4
(b) P (pa | s = bbb, F = 3) ∝ (1 − pa) 3 . The most probable value of pa (i.e.,
the value that maximizes the posterior probability density) is 0. The
mean value of pa is 1/5.
See figure 3.7b.
H0 is true H1 is true
pa = 1/6
0 50 100 150
1000/1
100/1
10/1
1/1
1/10
1/100
200
pa = 0.25
pa = 0.5
8
6
4
2
1000/1
100/1
10/1
8
6
4
2
1000/1
100/1
10/1
0
1/1 0
1/1
-2
-4
1/10
1/100
-2
-4
1/10
1/100
0 50 100 150 200 0 50 100 150 200
Solution to exercise 3.7 (p.54). The curves in figure 3.8 were found by finding
the mean and standard deviation of Fa, then setting Fa to the mean ± two
standard deviations to get a 95% plausible range for Fa, and computing the
three corresponding values of the log evidence ratio.
Solution to exercise 3.8 (p.57). Let Hi denote the hypothesis that the prize is
behind door i. We make the following assumptions: the three hypotheses H1,
H2 and H3 are equiprobable a priori, i.e.,
P (H1) = P (H2) = P (H3) = 1
. (3.36)
3
The datum we receive, after choosing door 1, is one of D = 3 and D = 2 (meaning
door 3 or 2 is opened, respectively). We assume that these two possible
outcomes have the following probabilities. If the prize is behind door 1 then
the host has a free choice; in this case we assume that the host selects at
random between D = 2 and D = 3. Otherwise the choice of the host is forced
and the probabilities are 0 and 1.
P (D = 2 | H1) = 1/2 P (D = 2 | H2) = 0 P (D = 2 | H3) = 1
P (D = 3 | H1) = 1/2 P (D = 3 | H2) = 1 P (D = 3 | H3) = 0
(3.37)
Now, using Bayes’ theorem, we evaluate the posterior probabilities of the
hypotheses:
P (Hi | D = 3) = P (D = 3 | Hi)P (Hi)
(3.38)
P (D = 3)
P (H1 | D = 3) = (1/2)(1/3)
P (D=3)
P (H2 | D = 3) = (1)(1/3)
P (D=3) P (H3 | D = 3) = (0)(1/3)
P (D=3)
(3.39)
The denominator P (D = 3) is (1/2) because it is the normalizing constant for
this posterior distribution. So
P (H1 | D = 3) = 1/3 P (H2 | D = 3) = 2/3 P (H3 | D = 3) = 0.
(3.40)
So the contestant should switch to door 2 in order to have the biggest chance
of getting the prize.
Many people find this outcome surprising. There are two ways to make it
more intuitive. One is to play the game thirty times with a friend and keep
track of the frequency with which switching gets the prize. Alternatively, you
can perform a thought experiment in which the game is played with a million
doors. The rules are now that the contestant chooses one door, then the game
Figure 3.8. Range of plausible
values of the log evidence in
favour of H1 as a function of F .
The vertical axis on the left is
log ; the right-hand
P (s | F,H1)
P (s | F,H0)
vertical axis shows the values of
P (s | F,H1)
P (s | F,H0) .
The solid line shows the log
evidence if the random variable
Fa takes on its mean value,
Fa = paF . The dotted lines show
(approximately) the log evidence
if Fa is at its 2.5th or 97.5th
percentile.
(See also figure 3.6, p.54.)