Information Theory, Inference, and Learning ... - MAELabs UCSD

Copyright Cambridge University Press 2003. On-screen viewing permitted. Printing not permitted. http://www.cambridge.org/0521642981
You can buy this book for 30 pounds or $50. See http://www.inference.phy.cam.ac.uk/mackay/itila/ for links.
540 45 — Gaussian Processes
45.2 From parametric models to Gaussian processes
Linear models
Let us consider a regression problem using H fixed basis functions, for example
one-dimensional radial basis functions as defined in equation (45.3).
Let us assume that a list of N input points {x (n) } has been specified and
define the N × H matrix R to be the matrix of values of the basis functions
{φh(x)} H h=1 at the points {xn},
Rnh ≡ φh(x (n) ). (45.17)
We define the vector yN to be the vector of values of y(x) at the N points,
yn ≡
Rnhwh. (45.18)
If the prior distribution of w is Gaussian with zero mean,
h
P (w) = Normal(w; 0, σ 2 wI), (45.19)
then y, being a linear function of w, is also Gaussian distributed, with mean
zero. The covariance matrix of y is
So the prior distribution of y is:
Q = 〈yy T 〉 = 〈Rww T R T 〉 = R 〈ww T 〉 R T
(45.20)
= σ 2 wRR T . (45.21)
P (y) = Normal(y; 0, Q) = Normal(y; 0, σ 2 w RRT ). (45.22)
This result, that the vector of N function values y has a Gaussian distribution,
is true for any selected points XN. This is the defining property of a
Gaussian process. The probability distribution of a function y(x) is a Gaussian
process if for any finite selection of points x (1) , x (2) , . . . , x (N) , the density
P (y(x (1) ), y(x (2) ), . . . , y(x (N) )) is a Gaussian.
Now, if the number of basis functions H is smaller than the number of
data points N, then the matrix Q will not have full rank. In this case the
probability distribution of y might be thought of as a flat elliptical pancake
confined to an H-dimensional subspace in the N-dimensional space in which
y lives.
What about the target values? If each target tn is assumed to differ by
additive Gaussian noise of variance σ2 ν from the corresponding function value
yn then t also has a Gaussian prior distribution,
P (t) = Normal(t; 0, Q + σ 2 νI). (45.23)
We will denote the covariance matrix of t by C:
C = Q + σ 2 νI = σ2 wRRT + σ 2 νI. (45.24)
Whether or not Q has full rank, the covariance matrix C has full rank since
σ2 νI is full rank.
What does the covariance matrix Q look like? In general, the (n, n ′ ) entry
of Q is
Qnn ′ = [σ2 wRR T ]nn ′ = σ2 w
h
φh(x (n) )φh(x (n′ ) ) (45.25)