Numerical Methods Contents - SAM

These results are an immediate consequence of the fact that
How to find suitable subspaces U k ?
∀A ∈ R n,n , A T = A ∃U ∈ R n,n , U −1 = U T : U T AU is diagonal.
→ linear algebra course & Chapter 5, Cor. 5.1.7.
Please note that for general regular M ∈ R n,n we cannot expect cond 2 (M) = κ(M).
Idea:
U k+1 ← U k + “ local steepest descent direction”
given by −gradJ(x (k) ) = b − Ax (k) = r k (residual → Def. 2.5.8)
U k+1 = Span {U k ,r k } , x (k) from (4.2.1). (4.2.2)
△
Obvious: r k = 0 ⇒ x (k) = x ∗ := A −1 b done ✔
✬
✩
4.2 Conjugate gradient method (CG)
Lemma 4.2.1 (r k ⊥ U k ).
With x (k) according to (4.2.1), U k from (4.2.2) the residual r k := b − Ax (k) satisfies
r T k u = 0 ∀u ∈ U k (”r k ⊥ U k ”).
Again we consider a linear system of equations Ax = b with s.p.d. (→ Def. 2.7.1) system matrix
A ∈ R n,n and given b ∈ R n .
Ôº¿ º¾
✫
Proof. Consider
ψ(t) = J(x (k) + tu) , u ∈ U k , t ∈ R .
✪
Ôº¿½ º¾
Liability of gradient method of Sect. 4.1.3:
NO MEMORY
1D line search in Alg. 4.1.4 is oblivious of former line searches, which rules out reuse of information
gained in previous steps of the iteration. This is a typical drawback of 1-point iterative methods.
By (4.2.1), t ↦→ ψ(t) has a global minimum in t = 0, which implies
dψ
dt (0) = grad J(x(k) ) T u = (Ax (k) − b) T u = 0 .
Since u ∈ U k was arbitrary, the lemma is proved.
✷
Idea:
Replace linear search with subspace correction
Given: initial guess x (0)
nested subspaces U 1 ⊂ U 2 ⊂ U 3 ⊂ · · · ⊂ U n = R n , dimU k = k
x (k) := argmin
x∈U k +x (0) J(x) , (4.2.1)
quadratic functional from (4.1.1)
Note: Once the subspaces U k and x (0) are fixed, the iteration (4.2.1) is well defined, because
J |Uk +x (0) always possess a unique minimizer.
✬
Corollary 4.2.2. If r l ≠ 0 for l = 0,...,k, k ≤ n, then {r 0 ,...,r k } is an orthogonal basis of
U k .
✫
Lemma 4.2.1 also implies that, if U 0 = {0}, then dimU k = k as long as x (k) ≠ x ∗ , that is, before
we have converged to the exact solution.
✩
✪
Obvious (from Lemma 4.1.2):
Thanks to (4.1.2), definition (4.2.1) ensures:
x (n) = x ∗ = A −1 b
∥
∥x (k+1) − x ∗∥ ∥ ∥A ≤
∥
∥x (k) − x ∗∥ ∥ ∥A
Ôº¿¼ º¾
(4.2.1) and (4.2.2) define the conjugate gradient method (CG) for the iterative solution of
Ax = b
(hailed as a “top ten algorithm” of the 20th century, SIAM News, 33(4))
Ôº¿¾ º¾