Chapter 3 (cont'd): Newton-Raphson, Secant, Fixed-Point Iteration

Chapter 3 (cont’d): Newton-Raphson, Secant,
Fixed-Point Iteration
Newton-Raphson Method
It is important to remember that for Newton-Raphson it is necessary to have a good initial guess, otherwise
the method may not converge.
Basic idea: Guess x 1 . Draw the tangent to f(x) at x 1 and use the intersection with the x-axis at x 2 as the
second guess. Repeat until convergence.
The gradient of the tangent line is simply f ′ (x n ) and so it has equation
To find x n+1 consider this at y = 0, i.e.,
giving
y − f(x n ) = f ′ (x n )(x − x n )
−f(x n ) = f ′ (x n )(x n+1 − x n )
x n+1 = x n − f(x n)
f ′ (x n )
Example: f(x) = xe x − 1 in the interval [0,1]
try first guess x 0 = 1,
x n+1 = x n − x ne xn − 1
(1 + x n )e xn
n x n f(x n ) f ′ (x n ) x n+1 e n+1 = x n+1 − x n
0 1.0 1.71828 5.43656 0.68394 -0.31606
1 0.68394 0.35534 3.33701 0.57745 -0.10649
2 0.57745 0.028721 2.81021 0.56723 -0.01021
Caution: If the initial guess is not accurate enough the algorithm may not converge to a root. This is
especially true when f ′ (x) becomes small somewhere (so called ‘shallow gradient’ functions).
General iterate:
Convergence of the Newton-Raphson method:
Define e n = x n − r.
x n+1 = x n − f(x n)
f ′ (x n )
e n+1 = e n − f(e n + r)
f ′ (e n + r) ≈ e n − f(r) + e nf ′ (r) + e2 n
2
f ′′ (r)
f ′ (r) + e n f ′′ (r) + e2 n
2
f ′′′ (r)
(Taylor)
We know that f(r) = 0 as r is a root.
Next use the expansion
1
1 + x = 1 − x + x2 + ..., |x| < 1
1

f(x)
r
x 2 x 1
x
Figure: Illustrating the Newton-Raphson algorithm.
f(x)
r
x 2 x 1
x
Figure: Illustrating the Newton-Raphson algorithm failing due to a point where f ′ (x) ≈ 0.
2

f(x)
x
r x 3
x 2 x 1
Figure: Illustrating the secant algorithm.
to give
To leading order then
( )(
e n+1 = e n − e n 1 + enf ′′ (r) f
2f ′ (r)
1 + e ′′ (r)
n
(
= e n − e n 1 + enf ′′ (r)
2f ′ (r)
= e2 n
2
f ′′ (r)
f ′ (r) + O(e3 n)
e n+1 = e2 n
2
f ′′ (r)
f ′ (r)
)
f ′ (r) + e2 f ′′′ −1
(r)
n f ′ (r)
f ′′′ (r)
f ′ (r) + e2 n
)(
1 − e n
f ′′ (r)
f ′ (r) − e2 n
as e n → ∞ provided that f ′ (r) ≠ 0.
)
f ′′ (r) 2
f ′ (r)
+ O(e 3 n) 2
since e n+1 /e 2 n is O(1) the Newton-Raphson method is said to converge QUADRATICALLY (p = 2).
Advantage: Quadratic convergence
Disadvantages: convergence not assured, can be computationally inefficient since the derivative f ′ (x) has
to be calculated at each time step.
The NR method is often used together with one of the bracketing methods discussed above. By using the
bisection method (for example) we can get a good initial guess and then obtain fast convergence with NR.
In order to avoid derivatives but still have a fast rate of converge (compared with linear) a commonly used
alternative is...
The Secant Method
(As seen in the shooting algorithm)
In the secant method the new iterate x n+1 is found as the intersection of the x-axis and the straight line
through f(x n ) and f(x n−1 ). The equation of the line through f(x n ) and f(x n−1 ) is given by
Imposing y = 0 at x = x n+1 we obtain
y − f(x n )
f(x n+1 ) − f(x n ) =
x − x n
x n−1 − x n
x n+1 = x n − (x n − x n−1 ) f(x n )
f(x n ) − f(x n−1 )
3
(∗)

• Notice (*) is equivalent to Newton-Raphson if we approximate
• It is better to implement (*) than
f ′ (x n ) ≈ f(x n) − f(x n−1 )
x n − x n−1
x n+1 = x n−1f(x n ) − x n f(x n−1 )
f(x n ) − f(x n−1 )
in order to reduce loss of significance errors. • A downside of the method, compared to Newton Raphson is
that we need TWO initial guesses now. BUT it is no longer necessary to compute derivatives.
How about convergence?
e n+1 + r = (en−1+r)f(en+r)−(en+r)f(en−1+r)
f(e n+r)−f(e n−1+r)
= r + en−1f(en+r)−enf(en−1+r)
f(e n+r)−f(e n−1+r)
Taylor expanding
e n+1 =
Now f(r) = 0 since r is a root, so
) (
)
e n−1
(f(r) + e n f ′ (r) + e2 n
2
f ′′ (r) − e n f(r) + e n−1 f ′ (r) + e2 n−1
2
f ′′ (r)
f(r) + e n f ′ (r) + e2 n
2
f ′′ (r) − f(r) − e n−1 f ′ (r) − e2 n−1
2
f ′′ (r)
e n+1 =
f
e ′′ (r)
n−1e n 2 (e n−e n−1)
(e n−e n−1)f ′ (r)+ f′′ (r)
2 (e 2 n −e2 n−1)
= en−1en
2
≈
en−1en
2
f ′′ (r)
f ′ (r)
f ′′ (r)
f ′ (r)
(
1 + 1 2 (e n + e n−1 ) f ′′ (r)
(
f ′ (r)
)
1 − 1 2 (e n + e n−1 ) f ′′ (r)
f ′ (r)
) −1
So to leading order
e n+1 ≈ 1 f ′′ (r)
2 f ′ (r) e n−1e n = Γe n−1 e n
as n → ∞
To determine the rate of convergence we need to find p such that
knowing that
e n+1
e n e n−1
∼ e n+1
(e n ) p as n → ∞
e n+1
e n e n−1
= Γ n
with Γ n → Γ as n → ∞
We look for a solution of the form
e n+1 = Ae p n, so that e n = Ae p n−1
Rearranging gives
( en
) 1/p
e n−1 =
A
which allows us to solve for A and p using
e n+1 = Γe n e n−1 =
Γ
e(1+1/p)
A1/p n = Ae p n
4

Equating the last terms gives
p 2 − p − 1 = 0, and Γ = A 1+1/p = A p
The rate of convergence p is therefore given by the positive root
and we have
p = 1 + √ 5
2
≈ 1.618....
e n+1
e p n
( 1
→
2
f ′′ ) 1/p
(r)
f ′ (r)
So the secant method converges almost quadratically, with p ≈ 1.618... and is therefore considerably faster
than the linear interval refinement methods examined earlier. Although slower to converge than Newton-
Raphson, it has the advantage of being much easier to implement.
Fixed Point Iteration
The final part of this section considers fixed point iteration.
Consider Newton-Raphson. This can be written as
x n+1 = G(x n ), where G(x n ) = x n − f(x n)
f ′ (x n ) .
We can think of this as a quite general algorithm, called fixed point iteration. It can be applied to the
problem f(x) = 0 any time you can write f(x) as f(x) = x − G(x), so that
f(x) = 0 is equivalent to
x = G(x).
Example: f(x) = x 2 − x − 2
(1) G(x) = x 2 − 2
(2) G(x) = √ x + 2
(3) G(x) = 1 + 2/x
(4) G(x) = x − (x 2 − x − 2)/m
G(x) is called the ITERATION FUNCTION.
The root can be looked for iteratively as
x n+1 = G(x n )
giving
(1) x n+1 = x 2 n − 2
(2) x n+1 = √ x n + 2
(3) x n+1 = 1 + 2/x n
(4) x n+1 = x n − (x 2 n − x n − 2)/m
NOTE: not all of these iterations will converge to the root! For the fixed point iteration to be useful we
have to check:
1. Given a starting point x 0 , it is possible to calculate iteratively x 1 ,x 2 ,... (consider e.g. G(x) = − √ x).
2. The sequence x 1 ,x 2 ,.... converges to some point r.
3. r is a FIXED POINT of G(x), i.e. G(r) = r.
5

f(x)
r
x
G(x)
r
x
Figure: Illustrating the fixed point iteration.
The point 1 is satisfied if G(x) : I → I maps the interval [a,b] onto itself. Then if x 0 ∈ I, implies that
and therefore that all subsequent x n+1 = G(x n ) ∈ I.
x 1 = G(x 0 ) ∈ I
Considering now point 3.
It is easy to show that if G(x) ∈ [a,b] then ∃ a fixed point such that G(r) = r.
In fact if G(a) = a or G(b) = b then a or b are fixed points.
If G(a) ≠ a or G(b) ≠ b then G(a) > a and G(b) < b, since G maps I → I.
So h(x) = G(x) − x is ∈ [a,b] and h(x) satisfies h(a) < 0 and h(b) > 0. By the intermediate value theorem,
therefore, ∃ a point r such that
h(r) = 0.
Therefore r is also a fixed point of G(x): that is G(r) = r.
Point 2 deals with the issue of convergence to a root. Consider this graphically
Cases (a) and (b) converge.
Cases (c) and (d) diverge.
The convergence of the iteration seems to depend on the slope of g(x) near to r, that is if |g ′ (x)| ≫ 1 we
have divergence.
In fact, examining the error,
e n+1 + r = G(e n + r)
≈ G(r) + e n G ′ (r)
e n+1 ≈ e n G ′ (r)
If |G ′ (r)| < 1 then the fixed point iteration CONVERGES LINEARLY.
If |G ′ (r)| > 0 (and |G ′ (r)| < 1) convergence is monotonic, otherwise oscillatory.
6

G(x)
(a)
G(x)
(b)
x
1
x 2
x 1
x 2
G(x)
x
r x0
x 0
(c)
G(x)
r
(d)
x
x
x 0 r r x0
x
Figure: Illustrating four examples of fixed point iteration. (a) and (b) converge, whereas (c) and (d) diverge.
7

Stopping Criteria
When you implement one of these methods remember to put a stopping criterion to end the iteration,
especially when the method is not guaranteed to converge. You can choose among
1. A bound in absolute value of the error
2. A bound in the relative value of the error
|x n+1 − x n | < δ (e.g. δ < 0.5 × 10 −6 for 6 dec. pl.)
|x n+1 − x n |
|x n+1 |
< δ
this is equivalent to a percentage error, but be careful with x n ∼ 0.
3. A check whether f(x n ) ∼ 0, i.e. ensure that
|f(x n )| < δ.
4. A limit on the number of steps in the iteration. (Always use this together with another condition
when there is a chance the method will not converge, to avoid an infinite loop.)
8