Best Approximation in the 2-norm

Then, we say f and g are orthogonal with respect to this inner product if ⟨f, g⟩ = 0.In general, an inner product on a vector space V over R, be it continuous or discrete, has thefollowing properties:1. ⟨f + g, h⟩ = ⟨f, h⟩ + ⟨g, h⟩ for all f, g, h ∈ V2. ⟨cf, g⟩ = c⟨f, g⟩ for all c ∈ R and all f ∈ V3. ⟨f, g⟩ = ⟨g, f⟩ for all f, g ∈ V4. ⟨f, f⟩ ≥ 0 for all f ∈ V, and ⟨f, f⟩ = 0 if and only if f = 0.This inner product can be used to define the norm of a function, which generalizes the conceptof the magnitude of a vector to functions, and therefore provides a measure of the “magnitude” ofa function. Recall that the magnitude of a vector v, denoted by ∥v∥, can be defined by∥v∥ = (v ⋅ v) 1/2 .Along similar lines, we define the 2-norm of a function f(x) defined on [a, b] by(∫ b1/2∥f∥ 2 = (⟨f, f⟩) 1/2 = [f(x)] dx) 2 .As we will see, it can be verified that this function does in fact satisfy the properties required of anorm. The continuous least-squares problem can then be described as the problem of findingsuch thatf n (x) =(∫ b∥f n − f∥ 2 =aan∑c j φ j (x)j=0) 1/2[f n (x) − f(x)] 2 dxis minimized. This minimization can be performed over C[a, b], the space of functions that arecontinuous on [a, b], but it is not necessary for a function f(x) to be continuous for ∥f∥ 2 to bedefined. Rather, we consider the space L 2 (a, b), the space of real-valued functions such that ∣f(x)∣ 2is integrable over (a, b).One very important property that ∥ ⋅ ∥ 2 has is that it satisfies the Cauchy-Schwarz inequality∣⟨f, g⟩∣ ≤ ∥f∥ 2 ∥g∥ 2 ,f, g ∈ V.This can be proven by noting that for any scalar c ∈ R,c 2 ∥f∥ 2 2 + 2c⟨f, g⟩ + ∥g∥ 2 2 = ∥cf + g∥ 2 2 ≥ 0.4

The left side is a quadratic polynomial in c. In order for this polynomial to not have any negativevalues, it must either have complex roots or a double real root. This is the case if the discrimantsatisfies4⟨f, g⟩ 2 − 4∥f∥ 2 2∥g∥ 2 2 ≤ 0,from which the Cauchy-Schwarz inequality immediately follows. By setting c = 1 and applying thisinequality, we immediately obtain the triangle-inequality property of norms.Suppose that we can construct a set of functions {φ j (x)} n j=0 that is orthogonal with respect tothe inner product of functions on [a, b]. That is,⟨φ k , φ j ⟩ =∫ baφ k (x)φ j (x) dx ={ 0 k ∕= jα k > 0 k = j .Then, the normal equations simplify to a trivial system[∫ b][φ k (x)] 2 dx c k =a∫ baφ k (x)f(x) dx,k = 0, 1, . . . , n,or, in terms of norms and inner products,∥φ k ∥ 2 2c k = ⟨φ k , f⟩,k = 0, 1, . . . , n.It follows that the coefficients {c j } n j=0 of the least-squares approximation f n(x) are simplyc k = ⟨φ k, f⟩∥φ k ∥ 2 , k = 0, 1, . . . , n.2If the constants {α k } n k=0 above satisfy α k = 1 for k = 0, 1, . . . , n, then we say that the orthogonalset of functions {φ j (x)} n j=0 is orthonormal. In that case, the solution to the continuous least-squaresproblem is simply given byc k = ⟨φ k , f⟩, k = 0, 1, . . . , n.Next, we will learn how sets of orthogonal polynomials can be computed.5