Karjalainen, Pasi A. Regularization and Bayesian methods for ...

28 2. Estimation theory
where I =] − ɛ,ɛ[× · · · ×] − ɛ,ɛ[⊂ R p and ɛ is small. So this cost gives zero penalty
if all components of the estimation error are small and a unit penalty if any of the
components is larger than ɛ. When C UC (θ, ˆθ) is substituted into the equation of
conditional Bayes cost (2.41) we obtain
∫
B UC (ˆθ|z) = p(θ|z)dθ (2.77)
˜θ∈I
∫
= 1 − p(θ|z)dθ (2.78)
where I states for the complement of I, and using the mean value theorem for
integrals [5] there is a value, say ˆθ, in I for which
˜θ∈I
B UC (ˆθ|z) = 1 − (2ɛ) p p(ˆθ|z) (2.79)
To minimize B UC (ˆθ|z) we must maximize p(ˆθ|z) so ˆθ UC can be defined by
p(ˆθ UC |z) ≥ p(ˆθ|z) (2.80)
for all ˆθ. Since ˆθ UC maximizes the posterior density of θ given the observations z,
ˆθ UC is also called the maximum a posteriori estimate ˆθ MAP .
ˆθ UC is the mode of density p(θ|z) and yet another name for the estimator
is the conditional mode estimator. It can be shown, that if we assume that the
prior distribution of θ is uniform in a region containing the maximum likelihood
estimate, then the maximum likelihood estimate is identical to maximum a posteriori
estimate, that is ˆθ ML = ˆθ MAP [133]. Clearly ˆθ MAP = ˆθ MS if the mode of the
density p(θ|z) equals to the mean η θ|z . This is the case when p(θ|z) is symmetric
and unimodal.
2.8 Linear minimum mean square estimator
In this section we restrict the form of the estimator to be a linear function of
data and derive the optimum estimator with this structural constraint. If certain
conditions for densities p(θ) and p(z) are fulfilled, this optimal linear estimator
turns out to be generally optimal.
Let the estimator be constrained to be a linear function of the data
ˆθ = Kz (2.81)
Let θ and z be random vectors with zero means and known covariances. No other
assumptions are made about the joint distribution of the parameters and data.
We derive the estimator that is of the form (2.81) and minimizes the mean square
Bayes cost B MS (ˆθ). We first note that
}
B MS (ˆθ) = E
{˜θT ˜θ
= E
{
(θ − ˆθ) T (θ − ˆθ)
}
(2.82)
(2.83)