Lesson 16 Linear Systems - Bruce E. Shapiro

Lesson 16
Linear Systems
In this section we will study the solution of a linear system of n equations with n
unknowns. We cover it briefly here because some understanding of the problem will
be necessary in our study of interpolation. However, this subject is normally part of
the Math 481A curriculum and hence will not be covered in any detail here.
Given a square n × n matrix A and n numbers b 1 , . . . , b n , we would like to solve the
linear system
Ax = b (16.1)
Since it is generally numerically inefficient to compute an inverse (it generally requires
O(n 3 ) operations we will not solve the system as
x = A −1 b (16.2)
although this is technically correct. Instead we will use the process of Gaussian
elimination. We begin by observing that if we can transform equation 16.1 into a
form
T x = b ′ (16.3)
where T is an upper triangular matrix, and b ′ is a modified version of b, then we can
read the solution for x n off the bottom row of the matrix, namely,
x n = b ′ n/T nn (16.4)
The matrix T is said to be in Row Echelon Form. The second to the last row of
the system 16.3 only depends on two variables, x n and x n−1 . Once we read off x n
then we can solve for x n−1 . This process of back substitution moves back up the
matrix one line at a time, solving for one variable at each step.
Gaussian elimination is then summarized as follows:
103

104 LESSON 16. LINEAR SYSTEMS
1. Convert the system Ax = b into an equivalent form T x = b ′ where T is uppertriangular.
2. Solve for x using back-substitution.
It is possible to take this idea one step further. If we can reduce equation 16.3 to the
form
Dx = b ′′ (16.5)
where D is a diagonal matrix, then it is even easier to read off the solutions, namel,
x i = b ′ i/D ii . In this revised form the matrix D is said to be in Reduced Row
Echelon Form, and the revised algorithm is called Gauss-Jordan Elimination.
The revised algorithm is summarized:
1. Convert the system Ax = b into an equivalent form T x = b ′ , where T is
upper-triangular.
2. Convert the system T x = b ′ into an equivalent form Dx = b ′′ , where D is
diagonal.
3. Solve for the x i .
We will outline the first algorithm (row reduction followed by back-substitution). We
start by writing the linear system
a 11 x 1 + a 12 x 2 + · · · + a 1n x n = b 1 (16.6)
a 21 x 1 + a 22 x 2 + · · · + a 2n x n = b 2 (16.7)
.
a n11 x 1 + a n2 x 2 + · · · + a nn x n = b n (16.8)
From equation 16.6 we can solve for x 1 in terms of x 2 , . . . , x n ,
x 1 = (b 1 − a 12 x 2 − a 13 x 3 − · · · − a 1n x n )/a 11 (16.9)
so if we already know x 2 , . . . , x n we can solve for x 1 immediately. But if we eliminate
x 1 from each of the remaining equations, we have system of n − 1 equations in the
n − 1 variables x 2 , . . . , x n , which is easier to solve than the original system because
it is smaller. We get this system by subtracting an appropriate multiple of the first
equation from each of the remaining equations, namely we subtract
(a i1 /a 11 ) × (a 11 x 1 + a 12 x 2 + · · · + a 1n x n = b 1 ) (16.10)
Math 481A
California State University Northridge
2008, B.E.Shapiro
Last revised: November 16, 2011

LESSON 16. LINEAR SYSTEMS 105
from the i th equation. The resulting system for x 2 , . . . , x n is
(a 22 − a 21 a 12 /a 11 )x 2 + · · · + (a 2n − a 21 a 1n /a 11 )x n = b 2 − a 21 b 1 /a 11 (16.11)
(a 32 − a 31 a 12 /a 11 )x 2 + · · · + (a 3n − a 31 a 1n /a 11 )x n = b 3 − a 31 b 1 /a 11 (16.12)
.
(a n2 − a n1 a 12 /a 11 )x 2 + · · · + (a nn − a n1 a 1n /a 11 )x n = b n − a n1 b 1 /a 11 (16.13)
The idea is to keep repeating this process until there is only one equation in the
reduced system. The result is an “upper triangular system.” If the original matrix
system is
⎛
⎜
⎝
⎞
a 11 a 12 a 13 · · · a 1n
a 21 a 22 a 23 · · · a 2n
a 31 a 32 a 33
.
. . .
⎟
. . ⎠
a n1 a n2 a n3 · · · a nn
Then the reduced matrix system is
⎛
⎜
⎝
⎞
x 1
x 2
x 3
⎟
. ⎠
a n1
⎛
=
⎜
⎝
⎞
b 1
b 2
b 3
⎟
. ⎠
x n
⎛
⎞ ⎛ ⎞ ⎛
a 11 a 12 a 13 · · · a ′ 1n x 1
0 a ′ 22 a ′ 23 · · · a ′ 2n
x 2
0 0 a ′ 33 · · · a ′ 3n
x 3
=
⎜
⎝ .
..
⎟ ⎜ ⎟ ⎜
. . ⎠ ⎝ . ⎠ ⎝
0 · · · 0 0 a ′ nn a n1
This process is called Gaussian Reduction. We can then solve the system by
starting on the bottom equation for x n , then the second from the bottom for x n−1 ,
and so forth, until we obtain x 1 . This second step is called back substitution.
Example 16.1. Solve the system
⎛
1 2
⎞ ⎛
3
⎝ 4 5 2 ⎠ ⎝
2 8 5
x
y
z
⎞
⎛
⎠ = ⎝
using Gaussian Reduction and back substitution.
5
10
15
⎞
⎠
⎞
b ′ 1
b ′ 2
b ′ 3
⎟
. ⎠
b‘ n
Solution. The first step is to subtract multiples of the first row from each of the
remaining two rows to make the coefficients of x zero in each of rows 2 and 3 of the
system. Since the coefficient of x is 1 in the first row, 4 in the second row, and 2 in
the third row, we subtract four times the first row from the second row, and twice
the first row from the third row.
⎛
⎝
1 2 3
4 − 4(1) 5 − 4(2) 2 − 4(3)
2 − 2(1) 8 − 2(2) 5 − 2(3)
⎞ ⎛
⎠ ⎝
x
y
z
⎞
⎛
⎠ = ⎝
5
10 − 4(5)
15 − 2(5)
⎞
⎠
2008, B.E.Shapiro
Last revised: November 16, 2011
Math 481A
California State University Northridge

106 LESSON 16. LINEAR SYSTEMS
⎛
⎝
1 2 3
0 −3 −10
0 4 −1
⎞ ⎛
⎠ ⎝
x
y
z
⎞
⎛
⎠ = ⎝
5
−10
5
Now the first column is all zeroes (except for the first row). The next step is to
subtract a multiple of the second row from the third row to get a zero in the second
entry of the third row. Since the coefficient of y is -3 in the second row and 4 in the
third row, we can add 4/3 times the second row to the third row.
⎛
⎝
1 2 3
0 −3 −10
0 4 + (4/3)(−3) −1 + (4/3)(−10)
⎛
⎝
1 2 3
0 −3 −10
0 0 −43/3
⎞ ⎛
⎠ ⎝
⎞ ⎛
⎠ ⎝
x
y
z
⎞
x
y
z
⎞
⎠ = ⎝
⎛
⎠ = ⎝
⎛
⎞
⎠
5
−10
−25/3
5
−10
5 + (4/3)(−10)
This completes the Gaussian elimination. We can then read off the solution by backsubstitution.
From the third row of the matrix,
From the second row of the matrix,
z = (−25/3)/(−43/3) = 25/43
−3y − 10z = −10
hence
y = − 1 60
(−10 + 10(25/43)) =
3 43
Finally, from the first row, we have
x + 2y + 3z = 5
( ) ( )
60 25
x = 5 − 2 − 3 = 20
43 43 43
We can write a simple recursive algorithm for Gaussian elimination as
Algorithm LinearSolve
Input: A, b
If n > 1,
{A ′ , b ′ } = Reduce(A, b)
LinearSolve (A ′ , b ′ )
End if
x 1 = (b 1 − a 12 x 2 − a 13 x 3 − · · · − a 1n x n )/a 11
⎞
⎠
⎞
⎠
Math 481A
California State University Northridge
2008, B.E.Shapiro
Last revised: November 16, 2011

LESSON 16. LINEAR SYSTEMS 107
Return {x 1 , x 2 , . . . , x n }
Algorithm Reduce
Input: A, b
n = dimension(b)
For k = 2, . . . , n,
m = a k1 /a 11
For j = 2, . . . , n,
a ′ k−1,j−1 = a kj − ma 1j
End For
b ′ k−1 = b k − mb 1
End For
Return {A ′ , b ′ }
The recurse algorithm can be almost literally translated into Mathematica:
reduce[A_, b_] := Module[{n, j, k, Aprime, bprime, m, row},
n = Length[b];
Aprime = {}; bprime = {};
For[k = 2, k n , k++,
m = A[[k, 1]]/A[[1, 1]];
row = {};
For[j = 2, j n, j++,
AppendTo[row, A[[k, j]] - m* A[[1, j]]];
];
AppendTo[Aprime, row];
AppendTo[bprime, b[[k]] - m*b[[1]]];
];
Return[{Aprime, bprime}];
];
gauss[A_, b_] := Module[{n, x, x1, Aprime, bprime},
n = Length[b];
x = {};
If[n > 1,
{Aprime, bprime} = reduce[A, b];
x = gauss[Aprime, bprime];
];
x1 = b[[1]]/A[[1, 1]];
For[k = 2, k n, k++,
x1 = x1 - A[[1, k]]x[[k - 1]]/A[[1, 1]];
];
2008, B.E.Shapiro
Last revised: November 16, 2011
Math 481A
California State University Northridge

108 LESSON 16. LINEAR SYSTEMS
x = Prepend[x, x1];
Return[x];
]
For example, to solve the system
⎛
⎞ ⎛ ⎞ ⎛ ⎞
0.116093 0.230616 0.34202 x 1 3
⎝0.461232 0.897598 1.28558⎠
⎝x 2
⎠ = ⎝17⎠ (16.14)
1.02606 1.92836 2.59808 x 3 5
One could use this function by typing
In:=
Out:=
A={{0.116093, 0.230616, 0.34202},
{0.461232, 0.897598, 1.28558},
{1.02606, 1.92836, 2.59808}};
b={3, 17, 5};
gauss[A, b]
{-33612.9, 27351.9, -7024.58}
In Mathematicawe can also solve the system directly by using the built in function
LinearSolve[A,b].
Gaussian elimination can fail if we divide by zero, and is susceptible to large errors or
possible overflow if we divide by a very small number (relative to the other numbers in
the matrix). Division occurs in two places in the algorithm: during the row reduction
phase where we define m = a k1 /a 11 and during the back-substitution step at the end
of the algorithm, where we solve for x 1 (here we also divide by a 11 , but its usually
a different a 11 ). These numbers are called pivots. The solution is to rearrange the
matrix (and the corresponding elements of b): if at any step along the way the pivot
is zero, then the entire row is exchanged with a row that does not have zero in that
column. If all of the remaining elements in that column are zero then the matrix is
singular and there is no unique solution (or no solution at all).
Math 481A
California State University Northridge
2008, B.E.Shapiro
Last revised: November 16, 2011