Karjalainen, Pasi A. Regularization and Bayesian methods for ...

30 2. Estimation theory
For the calculation of the mean we first have to form the equation for the posterior
density p(θ|z). First we note that the matrix inversion lemma [69] gives for the
inverse of the joint covariance of θ and z
(
) −1 (
)
C θ C θz C 11 C 12
=
(2.97)
C zθ C z C 21 C 22
where
C 11 = (C θ − C θz Cz
−1 C zθ ) −1 = C −1
θ
C 22 = (C z − C zθ C −1
θ
C 12 = C T 21 = −C 11 C θz C −1
z
C θz ) −1 = C −1
z
+ C −1
θ
C θz C 22 C zθ C −1
+ C −1
z
The density of z can be written in the form
{
p(z) ∝ exp − 1 ( ) ( )(
0 z T 0 0
2 0 Cz
−1
so that the posterior density p(θ|z) is obtained by forming
p(θ|z) = p(θ,z)
p(z)
{
∝
exp
{
= exp
{
= exp
− 1 2
− 1 2
(
θ
(2.98)
C zθ C 11 C θz Cz −1 (2.99)
= −C −1
θ
C θz C 22 (2.100)
0
z
) (
θ T z T C 11 C 12
C 21 C 22 − Cz
−1
)}
)(
θ
z
)}
(2.101)
(2.102)
(2.103)
(
θ T C 11 θ + 2θ T C 12 z + z T (C 22 − C −1
z )z )} (2.104)
− 1 2 (θT C 11 θ + 2θ T C 11 C θz C −1
z z (2.105)
+z T Cz
−1 C zθ C 11 C θz Cz −1 z)
{
= exp − 1 }
2 (θ − C θzCz
−1 z) T C 11 (θ − C θz Cz −1 z)
}
(2.106)
(2.107)
This is clearly a Gaussian density. The Gaussian conditional density is of the form
{
p(θ|z) ∝ exp − 1 }
2 (θ − E {θ|z})T C −1
θ|z (θ − E {θ|z}) (2.108)
Since ˆθ MS = E {θ|z}, C θ|z = C˜θMS
and comparing with (2.107) we can conclude
ˆθ MS = C θz C −1
z z (2.109)
C˜θMS
= C θ − C θz C −1
z C zθ (2.110)
that is exactly the linear minimum mean square estimator.