Information Theory, Inference, and Learning ... - MAELabs UCSD

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
502 41 — Learning as Inference
ψ(a, s 2 )
¡
¤ ¤ ¢£
¢
¢£§
¤ ¤
¤ ¢ § ¤
¡£¢
¢
¡£¢
(a)
¥¤
(a)
5
4
3
2
1
0
-1
-2
-3
¢ ¢
¦ § ¤ ¦ ¤ §
¡£¢
¨¨
© ©
0 1 2 3 4 5 6
Calculating the marginalized probability
¡£¢
(b)
¢£
¡
¢£¨
¢£ ©
¡¢ ¥¤ ¤ ¡£¢ ¦
¢£§
(b)
A
10
5
B
0
0 5 10
The output y(x; w) depends on w only through the scalar a(x; w), so we can
reduce the dimensionality of the integral by finding the probability density of
a. We are assuming a locally Gaussian posterior probability distribution over
w = wMP + ∆w, P (w | D, α) (1/ZQ) exp(− 1
2∆wTA∆w). For our single
neuron, the activation a(x; w) is a linear function of w with ∂a/∂w = x, so
for any x, the activation a is Gaussian-distributed.
⊲ Exercise 41.2. [2 ] Assuming w is Gaussian-distributed with mean w MP and
variance–covariance matrix A −1 , show that the probability distribution
of a(x) is
P (a | x, D, α) = Normal(aMP, s 2 ) =
1
√
2πs2 exp
− (a − aMP) 2
2s2
,
(41.28)
where aMP = a(x; wMP) and s2 = xTA−1x. This means that the marginalized output is:
P (t=1 | x, D, α) = ψ(aMP, s 2
) ≡ da f(a) Normal(aMP, s 2 ). (41.29)
This is to be contrasted with y(x; w MP) = f(a MP), the output of the most probable
network. The integral of a sigmoid times a Gaussian can be approximated
by:
ψ(a MP, s 2 ) φ(a MP, s 2 ) ≡ f(κ(s)a MP) (41.30)
with κ = 1/ 1 + πs 2 /8 (figure 41.10).
Demonstration
Figure 41.11 shows the result of fitting a Gaussian approximation at the optimum
w MP, and the results of using that Gaussian approximation and equa-
Figure 41.10. The marginalized
probability, and an approximation
to it. (a) The function ψ(a, s 2 ),
evaluated numerically. In (b) the
functions ψ(a, s 2 ) and φ(a, s 2 )
defined in the text are shown as a
function of a for s 2 = 4. From
MacKay (1992b).
Figure 41.11. The Gaussian
approximation in weight space
and its approximate predictions in
input space. (a) A projection of
the Gaussian approximation onto
the (w1, w2) plane of weight
space. The one- and
two-standard-deviation contours
are shown. Also shown are the
trajectory of the optimizer, and
the Monte Carlo method’s
samples. (b) The predictive
function obtained from the
Gaussian approximation and
equation (41.30). (cf. figure 41.2.)