Karjalainen, Pasi A. Regularization and Bayesian methods for ...

50 3. Regularization theory
can be very sensitive to errors in the data z even though the solutions were strictly
speaking continuous functions of the data.
The need to obtain stable solutions to ill-posed problems has led to several
methods in which the original problem is modified so that the solution of the
modified problem is near to the solution of the original problem but is less sensitive
to errors in the data. These methods are often called the regularization methods.
In regularization methods information about the desirable solution is used for the
modification of the original problem. In this sense the regularization approach is
related to the Bayesian approach to estimation.
Several regularization methods can be interpreted as stabilization of the inversion
of the matrix H T H in the least squares solution. Such methods are e.g.
Tikhonov regularization and principal component regression. Other approaches
to regularization are e.g. the use of truncated iterative methods, such as conjugate
gradient method, for solving linear systems of equations [78]. The truncated
singular value docomposition [75, 33] is also commonly used. All regularization
methods can be interpreted in the same formalism with the so-called filter factors,
see e.g. [182]. General references on regularization methods are e.g. [209, 72] or
[56] and especially [201] in connection to Bayes estimation.
3.2 Tikhonov regularization
We discuss here the most popular regularization method, Tikhonov regularization
method [208, 207] for solution of the least squares problem. We define here the
generalized Tikhonov regularized solution ˆθ α with equation
ˆθ α = arg min
θ
{‖L 1 (Hθ − z)‖ 2 + α 2 ‖L 2 (θ − θ ∗ )‖ 2 } (3.4)
where L T 1 L 1 = W is a positive definite matrix. The regularization parameter α
controls the weight given to minimization of the side constraint
Ω(θ) = ‖L 2 (θ − θ ∗ )‖ 2 (3.5)
relative to minimization of the weighted least squares index
l LS = ‖L 1 (Hθ − z)‖ 2 (3.6)
The term θ ∗ is the initial (prior) guess for the solution. In more common definitions
W = I, see e.g. [77].
The matrix L 2 is typically either the identity matrix I or a discrete approximation
D d of the d’th derivative operator. In this case L 2 is a banded matrix with
full row rank. For example the 1’st and 2’nd difference matrices are
⎛
⎞
1 −1 0 · · · 0
. 0 .. . .. . ..
. ..
D 1 =
⎜
(3.7)
⎝
.
. .. . .. . .. ⎟ 0 ⎠
0 · · · 0 1 −1