Truncated Gauss-Newton Method with Trust-region: an efficient ... - IFP

Truncated Gauss-Newton Method with
Trust-region:
an efficient solver for the non-linear
tomographic inverse problem
Frédéric Delbos
Jean-Charles Gilbert
Delphine Sinoquet

Objective
Where is my model solution ?
Develop a fast, accurate, robust and automatic
solver for the tomographic inverse problem
jerry

Outline
Preconditioned Conjugate Gradient Revisited
Trust-Region method - an alternative to line search
Singular model analysis

Terminology
Line search ambiguity:
The optimization method
The Gauss-Newton step failure
Gauss-Newton: GN ; index k
Trust-Region:T-R
Line Search:LS
Preconditioned Conjugate Gradient: PCG ; index i

Optimization method - GN
Non-linear cost function f
Linearization of T around
mk
Quadratic model F m of
k
f m
m
k

Optimization method - GN
Simplified expression of the quadratic function
f k
Value of in
Gradient of in
Approximated Hessian
(neglect second-order derivatives of traveltimes)
m
m
f k

Optimization method - GN

Preconditioning CG - Motivations
Solve the ill-conditioned linear problem:
Speed-up convergence of the CG algorithm

Preconditioning CG - Method
Transform linear system into:
1 1
k k k k
P H m P g
with P symmetric positive definite
and P as close as possible to H

Preconditioning CG - Method
Find a matrix such that
Minimize the quadratic function in new variables
with
Q
k
P Q Q
k
T
k k
T T 1
mQm, g Q
g , and H = Q H Q
k k k k k k k k

Preconditioning CG - Choice of P
Hessian decomposition into velocity and interface Blocks
with the block of H corresponding to velocity
and the block of H corresponding to interface
v
z
Hvi, vi
i
Hz , z
i
i i

Preconditioning CG - Choice of P
Hessian decomposition into a sum of block matrices
With D and L defined as:

Preconditioning CG - Choice of P
D is symmetric positive definite
We use a Cholesky factorization on matrix D:
The chosen preconditioners: (L. Chauvier et al. 2000)
Jacobi
D
L L
k D
T
D
Symmetric Gauss-Seidel
k k

Preconditioning CG - example
Number of CG iterations over GN iterations

Preconditioning CG - example
Relative residuals over CG iterations

Preconditioning CG - example
Non linear cost function over GN iterations

Preconditioning CG
Solve more accurately ill-conditioned problems
Speed-up convergence of the CG algorithm

Outline
Preconditioned Conjugate Gradient Revisited
Preconditioners
CG termination criteria
Trust-Region method - an alternative to line search
Singular model analysis

CG termination criteria - Choice
Objective: minimize the number of CG iterates and
obtained a sufficient reduction of the non linear cost
function
CG termination criteria:
Maximum number of CG iterations
Relative residual criterion
Final quadratic cost reduction criterion (QCR
experimental)

CG termination criteria - QCR
ni F
F F
, 1
k , k ,
ki i i

CG termination criteria - example
Number of PCG iterations for different eps of
the relative residual criterion

CG termination criteria - example
Non linear cost for different eps of
the relative residual criterion

CG termination criteria - Choice
Objective: minimize the number of CG iterates and
obtained a sufficient reduction of the non linear cost
function
CG termination criteria:
Maximum number of CG iterations
Relative residual criterion
Final quadratic cost reduction criterion (QCR
experimental)

Outline
Preconditioned Conjugate Gradient Revisited
Trust-Region method - an alternative to line search
Line search
Trust-region
Levenberg-Marquard
Singular model analysis

Minimize F
m :
k
LS method - Theory
Line Search subproblem to be solved at each GN
iteration with a CG algorithm
Acceptance of the step: Armijo criterion

LS method - movie

LS method - example
Non linear cost function over GN iterations
for a relative residual criterion at 1E-15

T-R method - Theory
Minimize F m in a region around the current iterate
k
Trust-region radius
K
Advantage: control the norm of the perturbation
Trust-region subproblem to be solved at each
GN iteration

T-R method - Theory
Solve the trust-region subproblem with Steihaug´s
method
* Solve the first order optimal condition with a preconditioned
conjugate gradient method (H s.p.d.)
* Ensuring that the computed step stays inside the trust-region
radius at each CG iteration
** Steihaug´s theorem: the P-norm of the perturbation
increases with CG iterates
** Global convergence
Generalized Cauchy point
at least better than the steepest descent method

T-R method - Theory
Concordance ratio K between the quadratic model
and the non linear cost function
Numerator actual reduction (non linear cost)
Denominator predicted reduction (quadratic cost)
Acceptation of the step following the value of K
If K
is negative or close to 0 Rejected step :
decrease K
and restart the step
If
is close to 1 Accepted step:
K
m m m
K1K

T-R method - Movie

T-R method - example
Comparison T-R/LS on the non linear
cost function over GN iterations

Levenberg-Marquard - Theory
The Levenberg-Marquard step is defined by the
C
resolution of the following subproblem:
H I mg
Equivalent to solve the trust-region minimization
subproblem: (Dennis & Schnabel 1983)
k
min F m
m
n
m
C
1
C C
with Hk
I gk
; is interpreted as a
Lagrange multiplier
TR allows an elegant and efficient choice of
C
k k
C

Outline
Preconditioned Conjugate Gradient Revisited
Trust-Region method - an alternative to line search
Line search
Trust-region
Levenberg-Marquard
Singular model analysis

Singular model analysis - discussion
H may be positive semi-definite
The reasons:
small regularization weights
ill-posed problem lack of a priori information
The main consequences:
slow or even no convergence of GN method
bad preconditioner: P also semi-positive definite
explosion of the final PCG perturbation in L2 norm
bad direction of the PCG perturbation g m
,
0

Singular model analysis - detection
Detection criteria of singular models
L2 norm of PCG perturbation
PCG perturbation angle
m K 2
g, m
Preconditioned norm of the generalized Cauchy point:
T 1
C C 1
C g P g
with S P g and
T 1 1
g P HP g
1/2 C
PS
2
L-curve find good regularization weight (S. Gomez, 2001)
Plot the parametrized curve:
* 2 *
J m mT
( mm) R( mm)
2
T
*

L-curve - example

Singular model analysis - remedy
Solve a new problem with more a priori information
constraint, stronger regularization, and/or new
parameterization
Find a low frequency solution (largest eigenvalues) of
the singular linear problem

Singular model analysis - low frequency
Choice of Trust-Region method: adds a priori
information by constraint on the perturbation norm
Work on PCG regularization properties: the best
determined components are computed in the first CG
iterates
Choosing weaker CG preconditioners (tune the CG
velocity convergence)
Modified Cholesky factorization
Convex combination
No preconditioning
T
P P E LL
P 1 I P
P
I
Choosing stronger CG stopping criteria (stop earlier
in the CG loop)
Criterion on the final reduction of the quadratic cost

Singular model analysis - Weaker
preconditioner
Non linear cost function: comparison of inversion between
the new convex preconditioner and without preconditioning

Singular model analysis - Weaker
preconditioner
Non linear cost function: comparison of inversion between
the new convex preconditioner and without preconditioning

Singular models analysis - QCR
Number of CG iterations: comparison between the final
quadratic cost reduction criterion and the relative residual criterion

Singular models analysis - QCR
Non linear cost function: comparison between the final
quadratic cost reduction criterion and the relative residual criterion

Conclusions
Trust-region method is:
More robust (solve singular models)
Automatic (choice of trust-region parameters)
PCG method:
Relevant PCG termination criterion
More flexibility of the preconditioner
Detect and solve singular model problems

Perspectives
Finalization of the work on unconstrained problems for
singular models
Optimization with linear constraints
Augmented Lagrangian: integration of the new
solver
Feasibility study of interior point method
(comparison with A.L.)