Karjalainen, Pasi A. Regularization and Bayesian methods for ...

46 2. Estimation theory
= ˆθ t−1 + K t ε t (2.276)
K t =
P t =
P t−1 ϕ t
ϕ T (2.277)
t P t−1 ϕ t + C vt
(
P t−1 ϕ t ϕ T )
t
I −
ϕ T P t−1 + C wt (2.278)
t P t−1 ϕ t + C vt
P t is then a recursive estimate of the covariance + C C˜θt wt
Kalman gain. If we now choose (make assumption)
and K t is called the
C vt = λ t (2.279)
C wt = ( λ −1
t − 1 ) ( P t−1 ϕ t ϕ T )
t
I −
ϕ T P t−1 (2.280)
t P t−1 ϕ t + C vt
then the Kalman filter reduces to the so-called recursive least squares (RLS) algorithm.
This leads to the conclusion, that the RLS is optimal in mean square
sense if the assumptions (2.279) and (2.280) are valid. Otherwise the RLS only
approximates the optimal recursive estimate. If λ t ≡ λ the method is called the
forgetting factor RLS. The forgertting factor RLS is popular e.g. is in time series
modeling, since its performance can be tuned with one parameter λ. Since λ is
usually tuned in RLS near to unity, the implicit assumption is, that C wt in Kalman
filter is “small” corresponding to slow variation of the model parameters.
With different choices of C wt , C vt and P 0 several algorithms of the form
ˆθ t = ˆθ t−1 + K t (z t − ϕ T t ˆθ t−1 ) (2.281)
ε t = z t − ϕ T t ˆθ t−1 (2.282)
can be obtained. The most popular forms are the normalized least mean square
(NLMS) algorithm
ˆθ t = ˆθ µϕ t
t−1 +
µϕ T t ϕ t + 1 ε t (2.283)
and the least mean square (LMS) algorithm
ˆθ t = ˆθ t−1 + µϕ t ε t (2.284)
where µ is the step size parameter that controls the convergence of the algorithm.
The connection of the LMS with the steepest descent method is clearly visible. The
connections of RLS, NLMS and LMS algorithms to Kalman filtering are discussed
in detail in [99]. The corresponding Kalman gain K t and momentary covariance
estimate P t are summarized in Table 2.1. These connections are fundamental,
when the implicit assumptions of the recursive algorithms are compared.
The time-varying algorithms presented here are all in their generic forms. In
many cases the effectiveness of the algorithm can be tuned with different matrix
decompositions and scalings during the iteration. A review of adaptive algorithms
can be found e.q. in [81]. Another common reference to recursive systems is [116].
The connection of the different computational forms of the adaptive algorithms