Numerical Methods Course Notes Version 0.1 (UCSD Math 174, Fall ...

9.1. LEAST SQUARES 117
The answer is not so enlightening as the means of finding the solution.
We should, for a moment, consider whether this is indeed the solution. Our calculations have
only shown an extrema at this choice of (a, b); could it not be a maxima?
9.1.3 Least Squares from Basis Functions
In many, but not all cases, the class of functions, F, is the span of a small set of functions. This
case is simpler to explore and we consider it here. In this case F can be viewed as a vector space
over the real numbers. That is, for f, g ∈ F, and α, β ∈ R, then αf + βg ∈ F, where the function
αf is that function such that (αf)(x) = αf(x).
Now let {g j (x)} m j=0
be a set of m + 1 linearly independent functions, i.e.,
c 0 g 0 (x) + c 1 g 1 (x) + . . . + c m g m (x) = 0 ∀x ⇒ c 0 = c 1 = . . . = c m = 0
Then we say that F is spanned by the functions {g j (x)} m j=0 if
⎧
⎫
⎨
F =
⎩ f(x) = ∑ ⎬
c j g j (x) | c j ∈ R, j = 0, 1, . . . , m
⎭ .
j
In this case the functions g j are basis functions for F. Note the basis functions need not be unique:
a given class of functions will usually have more than one choice of basis functions.
Example 9.1. The class F = {f(x) = ax + b | a, b ∈ R } is spanned by the two functions g 0 (x) = 1,
and g 1 (x) = x. However, it is also spanned by the two functions ˜g 0 (x) = 2x + 1, and ˜g 1 (x) = x − 1.
To find the least squares best approximant of F for a given set of data, we minimize the square
of the l 2 norm of the error; that is we minimize the function
⎡⎛
⎞ ⎤2
n∑
φ (c 0 , c 1 , . . . , c m ) = ⎣ c j g j (x k ) ⎠ − y k
⎦
(9.1)
k=0
Again we set partials to zero and solve
⎡⎛
0 = ∂φ n∑
= 2 ⎣⎝ ∑
c i
j
This can be rearranged to get
m∑
If we now let
j=0
[ ∑
k
d ij =
k=0
g j (x k )g i (x k )
⎝ ∑ j
⎞ ⎤
c j g j (x k ) ⎠ − y k
⎦ g i (x k )
]
c j =
n∑
g j (x k )g i (x k ), e i =
k=0
n∑
y k g i (x k )
k=0
n∑
y k g i (x k ),
Then we have reduced the problem to the linear system (again, called the normal equations):
⎡
⎤ ⎡ ⎤ ⎡ ⎤
d 00 d 01 d 02 · · · d 0m c 0 e 0
d 10 d 11 d 12 · · · d 1m
c 1
e 1
d 20 d 21 d 22 · · · d 2m
c 2
=
e 2
(9.2)
⎢
⎣
.
. . . ..
⎥ ⎢ ⎥ ⎢ ⎥
. ⎦ ⎣ . ⎦ ⎣ . ⎦
d m0 d m1 d m2 · · · d mm c m e m
k=0