Karjalainen, Pasi A. Regularization and Bayesian methods for ...

36 2. Estimation theory
Note, that the criterion (2.172) implies that the residual r = z −ẑ = z −H ˆθ LS
is orthogonal to all columns of the matrix H. In other words, r belongs to the null
space of H T that equals to the orthogonal complement of the range of H, that
is r ∈ N ( H ) T = R (H) ⊥ . By definition ẑ ∈ R (H) and dim (R (H)) = p. The
measurement z ∈ R M is now of the form
z = ẑ + r (2.175)
and we can see that ẑ is the orthogonal projection of z onto R (H), the linear
subspace spanned by the columns of the matrix H. We call these vectors the basis
vectors. This interpretation of the least squares problem is central in Chapter 6.
The generalized solution is obtained by multiplying the equation (2.167) with
a matrix L so that L T L = W
and with notations z ′ = Lz and H ′ = LH
Now the minimization of the generalized index
is obtained by using (2.174)
Lz = LHθ + Lv (2.176)
z ′ = H ′ θ + Lv (2.177)
l GLS = (z − Hθ)W(z − Hθ) T (2.178)
= (z ′ − H ′ θ)(z ′ − H ′ θ) T (2.179)
ˆθ GLS = (H ′T H ′ ) −1 H ′T z ′ (2.180)
= (H T L T LH) −1 H T L T Lz (2.181)
= (H T WH) −1 H T Wz (2.182)
This is seen to be equivalent to the Gauss–Markov estimate ˆθ GM if we choose
W = Cv −1 .
A classical reference for linear and nonlinear least squares problems is [110]
and for nonlinear optimization in general [93].
2.13 Comparison of ML, MAP and MS estimates
In this section we compare the maximum likelihood estimation with maximum a
posteriori estimation in case of Gaussian densities. Let the observation model be
z = h(θ) + v (2.183)
where θ and v are random parameters. Only the parameters θ are to be estimated.
Given θ, the observations z and the error v have the same density, except that the
mean E {z} = E {v} + h(θ). The density of z given θ is thus
p(z|θ) = p v (z − h(θ)|θ) (2.184)