Numerical Methods Contents - SAM

6.5 Non-linear Least Squares
If (6.5.3) is not satisfied, then the parameters are redundant in the sense that fewer parameters would
be enough to model the same dependence (locally at x ∗ ).
Example 6.5.1 (Non-linear data fitting (parametric statistics)).
Given:
Known:
data points (t i , y i ), i = 1, ...,m with measurements errors.
y = f(t,x) through a function f : R × R n ↦→ R depending non-linearly and smoothly on
parameters x ∈ R n .
Example: f(t) = x 1 + x 2 exp(−x 3 t), n = 3.
6.5.1 (Damped) Newton method
Φ(x ∗ ) = min ⇒ grad Φ(x) = 0, grad Φ(x) := ( ∂Φ
∂x 1
(x), ..., ∂Φ
∂x n
(x)) T ∈ R n .
Determine parameters by non-linear least squares data fitting:
x ∗ = argmin
x∈R n
∑ m |f(t i ,x) − y i | 2 = argmin 1
2 ‖F(x)‖ 2
i=1
x∈R n 2 , (6.5.1)
⎛
with F(x) = ⎝ f(t ⎞
1,x) − y 1
. ⎠ .
f(t m ,x) − y m
Ôº¾ º
✸
Simple idea: use Newton’s method (→ Sect. 3.4) to determine a zero of grad Φ : D ⊂ R n ↦→ R n .
Newton iteration (3.4.1) for non-linear system of equations grad Φ(x) = 0
Ôº¿½ º
chain rule (3.4.2) ➤ grad Φ(x) = DF(x) T F(x) ,
x (k+1) = x (k) − HΦ(x (k) ) −1 grad Φ(x (k) ) , (HΦ(x) = Hessian matrix) . (6.5.4)
Expressed in terms of F : R n ↦→ R n from (6.5.2):
Non-linear least squares problem
Given: F : D ⊂ R n ↦→ R m , m, n ∈ N, m > n.
Find: x ∗ ∈ D: x ∗ = argmin x∈D Φ(x) , Φ(x) := 2 1 ‖F(x)‖2 2 . (6.5.2)
Terminology: D ˆ= parameter space, x 1 , ...,x n ˆ= parameter.
As in the case of linear least squares problems (→ Rem. 6.0.3): a non-linear least squares problem
is related to an overdetermined non-linear system of equations F(x) = 0.
As for non-linear systems of equations (→ Chapter 3): existence and uniqueness of x ∗ in (6.5.2) has
to be established in each concrete case!
product rule (3.4.3) ➤ HΦ(x) := D(grad Φ)(x) = DF(x) T DF(x) +
This allows to rewrite (6.5.4) in concrete terms:
j=1
⇕
n∑ ∂ 2 F j
(HΦ(x)) i,k = (x)F
∂x
j=1 i ∂x j (x) + ∂F j
(x) ∂F j
(x) .
k ∂x k ∂x i
For Newton iterate x (k) : Newton correction s ∈ R n from LSE
⎛
⎞
m∑
⎝DF(x (k) ) T DF(x (k) ) + F j (x (k) )D 2 F j (x (k) ) ⎠
}
j=1
{{ }
=HΦ(x (k) )
m∑
F j (x)D 2 F j (x) ,
s = −DF(x (k) ) T F(x (k) ) . (6.5.5)
} {{ }
=grad Φ(x (k) )
✬
We require “independence for each parameter”:
∃ neighbourhood U(x ∗ )such that DF(x) has full rank n ∀ x ∈ U(x ∗ ) . (6.5.3)
(It means: the columns of the Jacobi matrix DF(x) are linearly independent.)
✩
Ôº¿¼ º
Remark 6.5.2 (Newton method and minimization of quadratic functional).
Newton’s method (6.5.4) for (6.5.2) can be read as successive minimization of a local quadratic
Ôº¿¾ º
✫
✪