Numerical Methods Contents - SAM

Algorithm 4.1.3 (Steepest descent).
(ger.: steilster Abstieg)
4.1.3 Gradient method for s.p.d. linear system of equations
Initial guess x (0) ∈ D, k = 0
repeat
d k := −gradF(x (k) )
t ∗ := argmin F(x (k) + td k ) ( line search)
t∈R
x (k+1) := x (k) + t ∗ d k
k := k + 1
∥
until ∥x (k) − x (k−1)∥ ∥
∥ ∥ ∥∥x ≤τ (k) ∥
d k ˆ= direction of steepest descent
linear search ˆ= 1D minimization:
use Newton’s method (→
Sect. 3.3.2.1) on derivative
a posteriori termination criterion,
see Sect. 3.1.2 for a discussion.
(τ ˆ= prescribed tolerance)
However, for the quadratic minimization problem (4.1.1) Alg. 4.1.3 will converge:
Adaptation: steepest descent algorithm Alg. 4.1.3 for quadratic minimization problem (4.1.1)
F(x) := J(x) = 2 1xT Ax − b T x ⇒ grad J(x) = Ax − b . (4.1.4)
This follows from A = A T , the componentwise expression
n∑ n∑
J(x) = 1 2 a ij x i x j − b i x i
i,j=1 i=1
and the definition (4.1.3) of the gradient.
➣
For the descent direction in Alg. 4.1.3 applied to the minimization of J from (4.1.1) holds
Ôº¿¿ º½
d k = b − Ax (k) =: r k the residual (→ Def. 2.5.8) for x (k−1) .
Alg. 4.1.3 for F = J from (4.1.1): function to be minimized in line search step:
ϕ(t) := J(x (k) + td k ) = J(x (k) ) + td T k (Ax(k) − b) + 1 2 t2 d T k Ad k ➙ a parabola ! .
dϕ
dt (t∗ ) = 0 ⇔
t ∗ = dT k d k
d T k Ad k
(unique minimizer) .
Ôº¿¿ º½
Note: d k = 0 ⇔ Ax (k) = b (solution found !)
The gradient (→ [40, Kapitel 7])
⎛ ⎞
∂F
∂x
⎜ i ⎟
grad F(x) = ⎝ . ⎠ ∈ R n (4.1.3)
∂F
∂x n
provides the direction of local steepest ascent/descent
of F
Note: A s.p.d. (→ Def. 2.7.1) ⇒ d T k Ad k > 0, if d k ≠ 0
ϕ(t) is a parabola that is bounded from below (upward opening)
Based on (4.1.4) and (4.1.3) we obtain the following steepest descent method for the minimization
problem (4.1.1):
Fig. 43
Of course this very algorithm can encouter plenty of difficulties:
Steepest descent iteration = gradient method for LSE Ax = b, A ∈ R n,n s.p.d., b ∈ R n :
Algorithm 4.1.4 (Gradient method).
• iteration may get stuck in a local minimum,
• iteration may diverge or lead out of D,
• line search may not be feasible.
Ôº¿¿ º½
Ôº¿¼ º½