First Semester in Numerical Analysis with Julia, 2020a

CHAPTER 5. APPROXIMATION THEORY 180
2. How do we find a, b that gives the line with the "best" approximation?
Observe that for each x i , there is the corresponding y i of the data point, and f(x i ) =
ax i + b, which is the predicted value by the linear approximation. We can measure error by
considering the deviations between the actual y coordinates and the predicted values:
(y 1 − ax 1 − b), (y 2 − ax 2 − b), ..., (y m − ax m − b)
There are several ways we can form a measure of error using these deviations, and each
approach gives a different line approximating the data. The best approximation means
finding a, b that minimizes the error measured in one of the following ways:
• E =max i {|y i − ax i − b|} ; minimax problem
• E = ∑ m
i=1 |y i − ax i − b|; absolute deviations
• E = ∑ m
i=1 (y i − ax i − b) 2 ; least squares problem
In this chapter we will discuss the least squares problem; the simplest one among the three
options. We want to minimize
E =
m∑
(y i − ax i − b) 2
i=1
with respect to the parameters a, b. For a minimum to occur, we must have
∂E
∂a
=0and
∂E
∂b =0.
We have:
m
∂E
∂a = ∑
i=1
∂E
∂b = m
∑
i=1
∂E
∂a (y i − ax i − b) 2 =
∂E
∂b (y i − ax i − b) 2 =
m∑
(−2x i )(y i − ax i − b) =0
i=1
m∑
(−2)(y i − ax i − b) =0
i=1
Using algebra, these equations can be simplified as
m∑ m∑
b x i + a x 2 i =
i=1
bm + a
i=1
m∑
x i =
i=1
m∑
x i y i
i=1
m∑
y i ,
i=1