Karjalainen, Pasi A. Regularization and Bayesian methods for ...

52 3. Regularization theory
where
L T L =
(
C −1
v 0
0 C −1
θ
)
(3.17)
Thus generalized Tikhonov regularization is the optimal mean square estimator if
v ∼ N(0,W −1 ) and θ ∼ N(θ ∗ ,α −2 (L T 2 L 2 ) −1 ).
Sometimes we do not have any particular assumptions about the parameters θ
themselves. They may not even have any physical meaning as in the case of generic
basis functions in linear least squares. More often we have beliefs about what the
measurements should be. In this case we have to regularize the prediction of the
model, that is, we want that Hθ has some desirable properties. The regularized
solution for W = I is then
ˆθ α = arg min
θ
{‖Hθ − z‖ 2 + α 2 ‖L 2 (Hθ − Hθ ∗ )‖ 2 } (3.18)
and the solution for this is clearly
ˆθ α = (H T H + α 2 H T L T 2 L 2 H) −1 (H T z + α 2 H T L T 2 L 2 Hθ ∗ ) (3.19)
This is of course of the same form than (3.15) with regularization matrix L ′ = L 2 H.
However in form (3.19) standard regularization matrices are directly applicable.
3.3 Principal component based regularization
In [23] it is emphasized that the notion of regularization has its origin in approximation
theory and many of the regularization methods there can be put in the
form
ˆθ = GH T z (3.20)
This estimator clearly covers the ordinary least squares solution with the choice
G = (H T H) −1 . The selection of G can thus be seen as approximation of the
inverse of H T H. The form (3.20) includes also the Tikhonov regularization. The
notion of ridge regression has been used for Tikhonov regularization with L = I
and the parameter α is called then the ridge parameter. In general ridge regression
problem we have some criterion, e.g. |θ| = c for some given c, for selecting the
parameter α [69].
It is possible to select
G = ∑ 1
v j vj T (3.21)
α j
j∈S
where v i and α i are the eigenvectors and eigenvalues of matrix H T H, respectively.
S is the index set S ⊂ I, where I = (1,..., rank (H T H)). The selection of the
index set S can be based on the eigenvalues α i or the correlation of the eigenvectors
with the data z. In the first approach the goal is to eliminate the large variances
in regression by deleting the components for which α i is small. This approach is
sometimes called the traditional principal component regression [23]. In the second
approach it is undesirable to delete components that have large correlations with