Karjalainen, Pasi A. Regularization and Bayesian methods for ...

26 2. Estimation theory
Now, since B(ˆθ|z) is a scalar,
B(ˆθ|z) = trace B(ˆθ|z) (2.54)
( {
}
= trace E θ T ∣
θ∣z
− 2ηθ|zˆθ T + ˆθ
)
T ˆθ (2.55)
where trace (A) is defined to be the sum of the diagonals of square matrix A. We
can use the identities [69]
trace (A + B) = trace (A) + trace (B) (2.56)
and then
trace (ABC) = trace (CAB) = trace (BCA) (2.57)
( ∣ }
B(ˆθ|z) = trace E
{θθ T ∣∣z
− 2ˆθη θ|z T + ˆθˆθ ) T (2.58)
(
= trace C θ|z + η θ|z ηθ|z T − 2ˆθη θ|z T + ˆθˆθ ) T (2.59)
(
)
= trace C θ|z + trace (ˆθ − η θ|z )(ˆθ − η θ|z ) T (2.60)
(
)
= trace C θ|z + trace (ˆθ − η θ|z ) T (ˆθ − η θ|z ) (2.61)
∥
∥ ∥∥
2
= trace C θ|z + ∥ˆθ − η θ|z (2.62)
The first term in right hand side of the equation does not depend on ˆθ(z) and is
clearly positive and the second can be made to zero by choosing ˆθ = η θ|z . Therefore
we conclude that the optimal Bayesian minimum mean square estimator is the
function η θ|z , that is, the conditional mean
ˆθ MS =
∫ ∞
−∞
θp(θ|z)dθ = E {θ|z} = η θ|z (2.63)
This result holds for all densities p(θ|z) [193]. The estimator ˆθ MS is sometimes
also called the conditional mean estimator. The expected value of the estimation
error ˜θ can be written as
}
}}
∣
E
{˜θ = E z
{E
{˜θ ∣z
(2.64)
= E z
{E
{θ − ˆθ
∣ }} ∣∣z
MS (2.65)
Now, since E{E{θ|z}|z} = E{θ|z}
{∫ ∞
∣ } ∣∣z
= E z θp(θ|z)dz − E
{ˆθMS } (2.66)
−∞
∣ }} ∣∣z
= E z
{ˆθMS − E
{ˆθMS (2.67)
}
E
{˜θ = 0 (2.68)