Information Theory, Inference, and Learning ... - MAELabs UCSD

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
42 2 — Probability, Entropy, and Inference
so
f(a) ≡
1 1 1 − e−a = + 1
1 + exp(−a) 2 1 + e−a = 1
2
= 1
(tanh(a/2) + 1).
2
(2.67)
ea/2 − e−a/2 ea/2 + 1
+ e−a/2 In the case b = log2 p/q, we can repeat steps (2.63–2.65), replacing e by 2,
to obtain
1
p = . (2.68)
1 + 2−b Solution to exercise 2.18 (p.36).
⇒
⇒ log
P (x | y) =
P (y | x)P (x)
P (y)
(2.69)
P (x = 1 | y)
P (x = 0 | y)
=
P (y | x = 1) P (x = 1)
P (y | x = 0) P (x = 0)
(2.70)
P (x = 1 | y)
P (x = 0 | y)
=
P (y | x = 1) P (x = 1)
log + log .
P (y | x = 0) P (x = 0)
(2.71)
Solution to exercise 2.19 (p.36). The conditional independence of d1 and d2
given x means
P (x, d1, d2) = P (x)P (d1 | x)P (d2 | x). (2.72)
This gives a separation of the posterior probability ratio into a series of factors,
one for each data point, times the prior probability ratio.
P (x = 1 | {di})
P (x = 0 | {di}) = P ({di} | x = 1) P (x = 1)
P ({di} | x = 0) P (x = 0)
(2.73)
= P (d1 | x = 1) P (d2 | x = 1) P (x = 1)
.
P (d1 | x = 0) P (d2 | x = 0) P (x = 0)
(2.74)
Life in high-dimensional spaces
Solution to exercise 2.20 (p.37). The volume of a hypersphere of radius r in
N dimensions is in fact
V (r, N) = πN/2
(N/2)! rN , (2.75)
but you don’t need to know this. For this question all that we need is the
r-dependence, V (r, N) ∝ r N . So the fractional volume in (r − ɛ, r) is
r N − (r − ɛ) N
r N
= 1 − 1 − ɛ
r
The fractional volumes in the shells for the required cases are:
N
N 2 10 1000
ɛ/r = 0.01 0.02 0.096 0.99996
ɛ/r = 0.5 0.75 0.999 1 − 2 −1000
. (2.76)
Notice that no matter how small ɛ is, for large enough N essentially all the
probability mass is in the surface shell of thickness ɛ.