Chapter 33 - Applied Mathematics - Illinois Institute of Technology
Chapter 33 - Applied Mathematics - Illinois Institute of Technology
Chapter 33 - Applied Mathematics - Illinois Institute of Technology
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
MATH 590: Meshfree Methods<br />
<strong>Chapter</strong> <strong>33</strong>: Adaptive Iteration<br />
Greg Fasshauer<br />
Department <strong>of</strong> <strong>Applied</strong> <strong>Mathematics</strong><br />
<strong>Illinois</strong> <strong>Institute</strong> <strong>of</strong> <strong>Technology</strong><br />
Fall 2010<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 1
Outline<br />
1 A Greedy Adaptive Algorithm<br />
2 The Faul-Powell Algorithm<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 2
The two adaptive algorithms discussed in this chapter both yield an<br />
approximate solution to the RBF interpolation problem.<br />
The algorithms have some similarity with some <strong>of</strong> the omitted material<br />
from <strong>Chapter</strong>s 21, 31 and 32.<br />
The contents <strong>of</strong> this chapter are based mostly on the papers<br />
[Faul and Powell (1999), Faul and Powell (2000),<br />
Schaback and Wendland (2000a), Schaback and Wendland (2000b)]<br />
and the book [Wendland (2005a)].<br />
As always, we concentrate on systems for strictly positive definite<br />
kernels (variations for strictly conditionally positive definite kernels also<br />
exist).<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 3
A Greedy Adaptive Algorithm<br />
One <strong>of</strong> the central ingredients is the use <strong>of</strong> the native space inner<br />
product discussed in <strong>Chapter</strong> 13.<br />
As always, we assume that our data sites are X = {x 1 , . . . , x N }.<br />
We also consider a second set Y ⊆ X .<br />
Let P Y f<br />
be the interpolant to f on Y ⊆ X .<br />
Then the first orthogonality lemma from <strong>Chapter</strong> 18 (with g = f ) yields<br />
〈f − P Y f<br />
, P Y f<br />
〉 NK (Ω) = 0.<br />
This leads to the energy split (see <strong>Chapter</strong> 18)<br />
‖f ‖ 2 N K (Ω) = ‖f − PY f<br />
‖ 2 N K (Ω) + ‖PY f<br />
‖ 2 N K (Ω) .<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 5
A Greedy Adaptive Algorithm<br />
We now consider an iteration on residuals.<br />
We pretend to start with our desired interpolant r 0 = Pf<br />
X<br />
set X .<br />
on the entire<br />
We also pick an appropriate sequence <strong>of</strong> sets Y k ⊆ X , k = 0, 1, . . . (we<br />
will discuss some possible heuristics for choosing these sets later).<br />
Then we iteratively define the residual functions<br />
r k+1 = r k − P Y k<br />
r k<br />
, k = 0, 1, . . . . (1)<br />
Remark<br />
In the actual algorithm below we will only deal with discrete vectors.<br />
Thus the vector r 0 will be given by the data values (since P f is<br />
supposed to interpolate f on X ).<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 6
A Greedy Adaptive Algorithm<br />
Now, the energy splitting identity with f = r k gives us<br />
‖r k ‖ 2 N K (Ω) = ‖r k − P Y k<br />
r k<br />
‖ 2 N K (Ω) + ‖PY k<br />
r k<br />
‖ 2 N K (Ω)<br />
(2)<br />
or, using the iteration formula (1),<br />
‖r k ‖ 2 N K (Ω) = ‖r k+1‖ 2 N K (Ω) + ‖r k − r k+1 ‖ 2 N K (Ω) . (3)<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 7
A Greedy Adaptive Algorithm<br />
We have the following telescoping sum for the partial sums <strong>of</strong> the norm<br />
<strong>of</strong> the residual updates P Y k<br />
r k<br />
:<br />
M∑<br />
k=0<br />
‖P Y k<br />
r k<br />
‖ 2 N K (Ω)<br />
(1)<br />
=<br />
(3)<br />
=<br />
M∑<br />
‖r k − r k+1 ‖ 2 N K (Ω)<br />
k=0<br />
M∑<br />
k=0<br />
{<br />
}<br />
‖r k ‖ 2 N K (Ω) − ‖r k+1‖ 2 N K (Ω)<br />
= ‖r 0 ‖ 2 N K (Ω) − ‖r M+1‖ 2 N K (Ω) ≤ ‖r 0‖ 2 N K (Ω) .<br />
Remark<br />
This estimate shows that the sequence <strong>of</strong> partial sums is monotone<br />
increasing and bounded, and therefore convergent — even for a poor<br />
choice <strong>of</strong> the sets Y k .<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 8
A Greedy Adaptive Algorithm<br />
If we can show that the residuals r k converge to zero, then we would<br />
have that the iteratively computed approximation<br />
u M+1 =<br />
M∑<br />
k=0<br />
P Y k<br />
r k<br />
=<br />
M∑<br />
(r k − r k+1 ) = r 0 − r M+1 (4)<br />
k=0<br />
converges to the original interpolant r 0 = P X f<br />
.<br />
Remark<br />
The omitted chapters contain iterative methods by which we<br />
approximate the interpolant by iterating an approximation method on<br />
the full data set.<br />
Here we are approximating the interpolant by iterating an interpolation<br />
method on nested (increasing) adaptively chosen subsets <strong>of</strong> the data.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 9
A Greedy Adaptive Algorithm<br />
Remark<br />
The present method also has some similarities with the (omitted)<br />
multilevel algorithms <strong>of</strong> <strong>Chapter</strong> 32.<br />
However,<br />
here: we compute the interpolant Pf<br />
X<br />
single kernel K<br />
on the set X based on a<br />
<strong>Chapter</strong> 32: the final interpolant is given as the result <strong>of</strong> using the<br />
spaces ⋃ M<br />
k=1 N K k<br />
(Ω), where K k is an appropriately scaled<br />
version <strong>of</strong> the kernel K .<br />
Moreover, the goal in <strong>Chapter</strong> 32 is to approximate f , not P f .<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 10
A Greedy Adaptive Algorithm<br />
To prove convergence <strong>of</strong> the residual iteration, we assume that we can<br />
find sets <strong>of</strong> points Y k such that at step k at least some fixed percentage<br />
<strong>of</strong> the energy <strong>of</strong> the residual is picked up by its interpolant, i.e.,<br />
with some fixed γ ∈ (0, 1].<br />
Then (3) and the iteration formula (1) imply<br />
‖P Y k<br />
r k<br />
‖ 2 N K (Ω) ≥ γ‖r k‖ 2 N K (Ω)<br />
(5)<br />
‖r k+1 ‖ 2 N K (Ω) = ‖r k‖ 2 N K (Ω) − ‖PY k<br />
r k<br />
‖ 2 N K (Ω) ,<br />
and therefore<br />
‖r k+1 ‖ 2 N K (Ω) ≤ ‖r k‖ 2 N K (Ω) − γ‖r k‖ 2 N K (Ω) = (1 − γ)‖r k‖ 2 N K (Ω) .<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 11
Applying the bound<br />
A Greedy Adaptive Algorithm<br />
recursively yields<br />
‖r k+1 ‖ 2 N K (Ω) ≤ (1 − γ)‖r k‖ 2 N K (Ω)<br />
Theorem<br />
If the choice <strong>of</strong> sets Y k satisfies ‖P Y k<br />
r k<br />
‖ 2 N K (Ω) ≥ γ‖r k‖ 2 N K (Ω), then the<br />
residual iteration (see (4))<br />
u M =<br />
M−1<br />
∑<br />
k=0<br />
P Y k<br />
r k<br />
= r 0 − r M , r k+1 = r k − P Y k<br />
r k<br />
, k = 0, 1, . . .<br />
converges linearly in the native space norm. After M steps <strong>of</strong> iterative<br />
refinement there is an error bound<br />
‖P X f − u M ‖ 2 N K (Ω) = ‖r 0 − u M ‖ 2 N K (Ω) = ‖r M‖ 2 N K (Ω) ≤ (1 − γ)M ‖r 0 ‖ 2 N K (Ω) .<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 12
A Greedy Adaptive Algorithm<br />
Remark<br />
This theorem has various limitations:<br />
The norm involves the kernel K which makes it difficult to find sets<br />
Y k that satisfy (5).<br />
The native space norm <strong>of</strong> the initial residual r 0 is not known.<br />
A way around these problems is to use an equivalent discrete norm on<br />
the set X .<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 13
A Greedy Adaptive Algorithm<br />
Schaback and Wendland establish an estimate <strong>of</strong> the form<br />
‖r 0 − u M ‖ 2 X ≤ C c<br />
(1 − δ c2<br />
C 2 ) M/2<br />
‖r 0 ‖ 2 X ,<br />
where c and C are constants denoting the norm equivalence, i.e.,<br />
c‖u‖ X ≤ ‖u‖ NK (Ω) ≤ C‖u‖ X<br />
for any u ∈ N K (Ω), and where δ is a constant analogous to γ (but<br />
based on use <strong>of</strong> the discrete norm ‖ · ‖ X in (5)).<br />
In fact, any discrete l p norm on X can be used.<br />
In the implementation below we will use the maximum norm.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 14
A Greedy Adaptive Algorithm<br />
In [Schaback and Wendland (2000b)] a basic version <strong>of</strong> this<br />
algorithm — where the sets Y k consist <strong>of</strong> a single point — is<br />
described and tested.<br />
The resulting approximation yields the best M-term approximation<br />
to the interpolant.<br />
Remark<br />
This idea is related to the concepts <strong>of</strong><br />
greedy approximation algorithms (see, e.g., [Temlyakov (1998)])<br />
and<br />
sparse approximation (see, e.g., [Girosi (1998)]).<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 15
A Greedy Adaptive Algorithm<br />
If the set Y k consists <strong>of</strong> only a single point y k , then the partial<br />
interpolant P Y k<br />
r k<br />
is particularly simple:<br />
with<br />
P Y k<br />
r k<br />
= βK (·, y k )<br />
β = r k(y k )<br />
K (y k , y k )<br />
This follows immediately from the usual RBF expansion (which<br />
consists <strong>of</strong> only one term here) and the interpolation condition<br />
P Y k<br />
r k<br />
(y k ) = r k (y k ).<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 16
A Greedy Adaptive Algorithm<br />
The point y k is picked to be the point in X where the residual is<br />
largest, i.e.,<br />
|r k (y k )| = ‖r k ‖ ∞ .<br />
This choice <strong>of</strong> “set” Y k certainly satisfies the constraint (5):<br />
‖P Y k<br />
r k<br />
‖ 2 N K (Ω)<br />
= ‖βK (·, y k )‖ 2 N K (Ω)<br />
=<br />
r k (y k )<br />
∥K (y k , y k ) K (·, y k)<br />
∥<br />
2<br />
N K (Ω)<br />
≤ γ‖r k ‖ 2 N K (Ω) , 0 < γ ≤ 1.<br />
Here we require K (·, y k ) ≤ K (y k , y k ) which is certainly true for<br />
positive definite translation invariant kernels (cf. <strong>Chapter</strong> 3).<br />
However, in general we only know that<br />
|K (x, y)| 2 ≤ K (x, x)K (y, y) (see<br />
[Berlinet and Thomas-Agnan (2004)]).<br />
The interpolation problem is (approximately) solved without having<br />
to invert any linear systems.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 17
A Greedy Adaptive Algorithm<br />
Algorithm (Greedy one-point)<br />
Input data locations X , associated values f <strong>of</strong> f , tolerance tol > 0<br />
Set initial residual r 0 = P X f<br />
| X = f , initialize u 0 = 0, e = ∞, k = 0<br />
Choose starting point y k ∈ X<br />
While e > tol do<br />
Set β = r k(y k )<br />
K (y k , y k )<br />
For 1 ≤ i ≤ N do<br />
r k+1 (x i ) = r k (x i ) − βK (x i , y k )<br />
u k+1 (x i ) = u k (x i ) + βK (x i , y k )<br />
end<br />
Find e = max |r k+1| and the point y k+1 where it occurs<br />
X<br />
Increment k = k + 1<br />
end<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 18
A Greedy Adaptive Algorithm<br />
Remark<br />
It is important to realize that in our MATLAB implementation we<br />
never actually compute the initial residual r 0 = P X f<br />
.<br />
All we require are the values <strong>of</strong> r 0 on the grid X <strong>of</strong> data sites.<br />
However, since Pf<br />
X | X = f | X the values r 0 (x i ) are given by the<br />
interpolation data f (x i ) (see line 5 <strong>of</strong> the code).<br />
Moreover, since the sets Y k are subsets <strong>of</strong> X the value r k (y k )<br />
required to determine β is actually one <strong>of</strong> the current residual<br />
values (see line 10 <strong>of</strong> the code).<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 19
A Greedy Adaptive Algorithm<br />
Remark<br />
We use DistanceMatrix together with rbf to compute both<br />
K (y k , y k ) (on lines 9 and 10) and<br />
K (x i , y k ) needed for the updates <strong>of</strong> the residual r k+1 and the<br />
approximation u k+1 on lines 11–14.<br />
Note that the “matrices” DM_data, IM, DM_res, RM, DM_eval,<br />
EM are only column vectors since only one center, y k , is involved.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 20
A Greedy Adaptive Algorithm<br />
Remark<br />
The algorithm demands that we compute the residuals r k on the<br />
data sites.<br />
The partial approximants u k to the interpolant can be evaluated<br />
anywhere.<br />
If we do this also at the data sites, then we are required to use a<br />
plotting routine that differs from our usual one (such as trisurf<br />
built on a triangulation <strong>of</strong> the data sites obtained with the help <strong>of</strong><br />
delaunayn).<br />
We instead follow the same procedure as in all <strong>of</strong> our other<br />
programs, i.e., to evaluate u k on a 40 × 40 grid <strong>of</strong> equally spaced<br />
points. This has been implemented on lines 11–15 <strong>of</strong> the program.<br />
Note that the updating procedure has been vectorized in MATLAB<br />
allowing us to avoid the for-loop over i in the algorithm.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 21
A Greedy Adaptive Algorithm<br />
Program (RBFGreedyOnePoint2D.m)<br />
1 rbf = @(e,r) exp(-(e*r).^2); ep = 5.5;<br />
2 N = 16641; dsites = CreatePoints(N,2,’h’);<br />
3 neval = 40; epoints = CreatePoints(neval^2,2,’u’);<br />
4 tol = 1e-5; kmax = 1000;<br />
5 res = testfunctionsD(dsites); u = 0;<br />
6 k = 1; maxres(k) = 999999;<br />
7 ykidx = (N+1)/2; yk(k,:) = dsites(ykidx,:);<br />
8 while (maxres(k) > tol && k < kmax)<br />
9 DM_data = DistanceMatrix(yk(k,:),yk(k,:));<br />
10 IM = rbf(ep,DM_data); beta = res(ykidx)/IM;<br />
11 DM_res = DistanceMatrix(dsites,yk(k,:));<br />
12 RM = rbf(ep,DM_res);<br />
13 DM_eval = DistanceMatrix(epoints,yk(k,:));<br />
14 EM = rbf(ep,DM_eval);<br />
15 res = res - beta*RM; u = u + beta*EM;<br />
16 [maxres(k+1), ykidx] = max(abs(res));<br />
17 yk(k+1,:) = dsites(ykidx,:); k = k + 1;<br />
18 end<br />
19 exact = testfunctionsD(epoints);<br />
20 rms_err = norm(u-exact)/neval<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 22
A Greedy Adaptive Algorithm<br />
To illustrate the greedy one-point algorithm we perform two<br />
experiments.<br />
Both tests use data obtained by sampling Franke’s function at 16641<br />
Halton points in [0, 1] 2 .<br />
Test 1 is based on Gaussians,<br />
Test 2 uses inverse multiquadrics.<br />
For both tests we use the same shape parameter ε = 5.5.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 23
A Greedy Adaptive Algorithm<br />
Figure: 1000 selected points and residual for greedy one point algorithm with<br />
Gaussian RBFs and N = 16641 data points.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 24
A Greedy Adaptive Algorithm<br />
Figure: Fits <strong>of</strong> Franke’s function for greedy one point algorithm with Gaussian<br />
RBFs and N = 16641 data points. Top left to bottom right: 1 point, 2 points, 4<br />
points, final fit with 1000 points.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 25
A Greedy Adaptive Algorithm<br />
Remark<br />
In order to obtain our approximate interpolants we used<br />
a tolerance <strong>of</strong> 10 −5<br />
along with an additional upper limit <strong>of</strong> kmax=1000 on the number <strong>of</strong><br />
iterations.<br />
For both tests the algorithm uses up all 1000 iterations.<br />
The final maximum residual is<br />
maxres = 0.0075 for Gaussians, and<br />
maxres = 0.0035 for inverse MQs.<br />
In both cases there occurred several multiple point selections.<br />
Contrary to interpolation problems based on the solution <strong>of</strong> a<br />
linear system, multiple point selections do not pose a problem<br />
here.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 26
A Greedy Adaptive Algorithm<br />
Figure: 1000 selected points and residual for greedy one point algorithm with<br />
IMQ RBFs and N = 16641 data points.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 27
A Greedy Adaptive Algorithm<br />
Figure: Fits <strong>of</strong> Franke’s function for greedy one point algorithm with IMQ<br />
RBFs and N = 16641 data points. Top left to bottom right: 1 point, 2 points, 4<br />
points, final fit with 1000 points.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 28
A Greedy Adaptive Algorithm<br />
Remark<br />
We note that the inverse multiquadrics have a more global<br />
influence than the Gaussians (for the same shape parameter).<br />
This effect is clearly evident in the first few approximations to the<br />
interpolants in the figures.<br />
From the last figure we see that the greedy algorithm enforces<br />
interpolation <strong>of</strong> the data only on the most recent set Y k (i.e., for<br />
the one-point algorithm studied here only at a single point).<br />
If one wants to maintain the interpolation achieved in previous<br />
iterations, then the sets Y k should be nested.<br />
This, however, would have a significant effect on the execution<br />
time <strong>of</strong> the algorithm since the matrices at each step would<br />
increase in size.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 29
A Greedy Adaptive Algorithm<br />
Remark<br />
One advantage <strong>of</strong> this very simple algorithm is that no linear<br />
systems need to be solved.<br />
This allows us to approximate the interpolants for large data sets<br />
even for globally supported kernels,<br />
and also with small values <strong>of</strong> ε (and therefore an associated<br />
ill-conditioned interpolation matrix).<br />
One should not expect too much in this case, however, as the<br />
results in the following figure show where we used a value <strong>of</strong><br />
ε = 0.1 for the shape parameter.<br />
A lot <strong>of</strong> smoothing occurs so that the convergence to the RBF<br />
interpolant is very slow.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 30
A Greedy Adaptive Algorithm<br />
Figure: 1000 selected points (only 20 <strong>of</strong> them distinct) and fit <strong>of</strong> Franke’s<br />
function for greedy one point algorithm with flat Gaussian RBFs (ε = 0.1) and<br />
N = 16641 data points.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 31
A Greedy Adaptive Algorithm<br />
Remark<br />
In the pseudo-code <strong>of</strong> the algorithm matrix-vector multiplications<br />
are not required.<br />
However, MATLAB allows for a vectorization <strong>of</strong> the for-loop which<br />
does result in two matrix-vector multiplications.<br />
For practical situations, e.g., for smooth kernels and densely<br />
distributed points in X the convergence can be rather slow.<br />
The simple greedy algorithm described above is extended in<br />
[Schaback and Wendland (2000b)] to a version that adaptively<br />
uses kernels <strong>of</strong> varying scales.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 32
The Faul-Powell Algorithm<br />
Another iterative algorithm was suggested in<br />
[Faul and Powell (1999), Faul and Powell (2000)].<br />
From our earlier discussions we know that it is possible to express a<br />
kernel interpolant in terms <strong>of</strong> cardinal functions uj ∗ , j = 1, . . . , N, i.e.,<br />
P f (x) =<br />
N∑<br />
f (x j )uj ∗ (x).<br />
j=1<br />
The basic idea <strong>of</strong> the Faul-Powell algorithm is to use approximate<br />
cardinal functions Ψ j instead.<br />
Of course, this will only give an approximate value for the interpolant,<br />
and therefore an iteration on the residuals is suggested to improve the<br />
accuracy <strong>of</strong> this approximation.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 34
The Faul-Powell Algorithm<br />
The approximate cardinal functions Ψ j , j = 1, . . . , N, are determined as<br />
linear combinations <strong>of</strong> the basis functions K (·, x l ) for the interpolant,<br />
i.e.,<br />
Ψ j = ∑ l∈L j<br />
b jl K (·, x l ), (6)<br />
where L j is an index set consisting <strong>of</strong> n (n ≈ 50) indices that are used<br />
to determine the approximate cardinal function.<br />
Example<br />
The n nearest neighbors <strong>of</strong> x j will usually do.<br />
Remark<br />
The basic philosophy <strong>of</strong> this algorithm is very similar to that <strong>of</strong> the<br />
omitted fixed level iteration <strong>of</strong> <strong>Chapter</strong> 31 where approximate MLS<br />
generating functions were used as approximate cardinal functions.<br />
The Faul-Powell algorithm can be interpreted as a Krylov<br />
subspace method.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 35
The Faul-Powell Algorithm<br />
Remark<br />
In general, the choice <strong>of</strong> index sets allows much freedom, and this<br />
is the reason why we include the algorithm in this chapter on<br />
adaptive iterative methods.<br />
As pointed out at the end <strong>of</strong> this section, there is a certain duality<br />
between the Faul-Powell algorithm and the greedy algorithm <strong>of</strong> the<br />
previous section.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 36
The Faul-Powell Algorithm<br />
For every j = 1, . . . , N, the coefficients b jl are found as solution <strong>of</strong> the<br />
(relatively small) n × n linear system<br />
Ψ j (x i ) = δ jk , i ∈ L j . (7)<br />
These approximate cardinal functions are computed in a<br />
pre-processing step.<br />
In its simplest form the residual iteration can be formulated as<br />
u (0) (x) =<br />
N∑<br />
f (x j )Ψ j (x)<br />
j=1<br />
u (k+1) (x) = u (k) (x) +<br />
N∑<br />
j=1<br />
[<br />
]<br />
f (x j ) − u (k) (x j ) Ψ j (x), k = 0, 1, . . . .<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 37
The Faul-Powell Algorithm<br />
Instead <strong>of</strong> adding the contribution <strong>of</strong> all approximate cardinal functions<br />
at the same time, this is done in a three-step process in the<br />
Faul-Powell algorithm.<br />
To this end, we choose index sets L j , j = 1, . . . , N − n, such that<br />
while making sure that j ∈ L j .<br />
L j ⊆ {j, j + 1, . . . , N}<br />
Remark<br />
If one wants to use this algorithm to approximate the interpolant based<br />
on conditionally positive definite kernels <strong>of</strong> order m, then one needs to<br />
ensure that the corresponding centers form an (m − 1)-unisolvent set<br />
and append a polynomial to the local expansion (6).<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 38
Step 1<br />
The Faul-Powell Algorithm<br />
We define u (k)<br />
0<br />
= u (k) , and then iterate<br />
u (k)<br />
j<br />
= u (k)<br />
j−1 + θ(k) j<br />
Ψ j , j = 1, . . . , N − n, (8)<br />
with<br />
θ (k)<br />
j<br />
= 〈P f − u (k)<br />
j−1 , Ψ j〉 NK (Ω)<br />
. (9)<br />
〈Ψ j , Ψ j 〉 NK (Ω)<br />
Remark<br />
The stepsize θ (k)<br />
j<br />
is chosen so that the native space best<br />
approximation to the residual P f − u (k)<br />
j−1<br />
from the space spanned by the<br />
approximate cardinal functions Ψ j is added.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 39
Step 1 (cont.)<br />
The Faul-Powell Algorithm<br />
Using the representation<br />
Ψ j = ∑ l∈L j<br />
b jl K (·, x l ),<br />
the reproducing kernel property <strong>of</strong> K , and the (local) cardinality<br />
property Ψ j (x i ) = δ jk , i ∈ L j we can calculate the denominator <strong>of</strong> (9) as<br />
〈Ψ j , Ψ j 〉 NK (Ω) = 〈Ψ j , ∑ l∈L j<br />
b jl K (·, x l )〉 NK (Ω)<br />
= ∑ l∈L j<br />
b jl 〈Ψ j , K (·, x l )〉 NK (Ω)<br />
= ∑ l∈L j<br />
b jl Ψ j (x l ) = b jj<br />
since we have j ∈ L j by construction <strong>of</strong> the index set L j .<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 40
Step 1 (cont.)<br />
The Faul-Powell Algorithm<br />
Similarly, we get for the numerator<br />
〈P f − u (k)<br />
j−1 , Ψ j〉 NK (Ω) = 〈P f − u (k)<br />
j−1 , ∑ l∈L j<br />
b jl K (·, x l )〉 NK (Ω)<br />
= ∑ l∈L j<br />
b jl 〈P f − u (k)<br />
j−1 , K (·, x l)〉 NK (Ω)<br />
= ∑ ( )<br />
b jl P f − u (k) (x l )<br />
l∈L j<br />
j−1<br />
= ∑ (<br />
)<br />
b jl f (x l ) − u (k)<br />
j−1 (x l) .<br />
l∈L j<br />
Therefore (8) and (9) can be written as<br />
u (k)<br />
j<br />
= u (k)<br />
j−1 + Ψ ∑ (<br />
)<br />
j<br />
b jl f (x l ) − u (k)<br />
b<br />
j−1 (x l) , j = 1, . . . , N − n.<br />
jj<br />
l∈L j<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 41
Step 2<br />
The Faul-Powell Algorithm<br />
Next we interpolate the residual on the remaining n points (collected<br />
via the index set L ∗ ).<br />
Thus, we find a function v (k) in span{K (·, x j ) : j ∈ L ∗ } such that<br />
v (k) (x i ) = f (x i ) − u (k)<br />
N−n (x i), i ∈ L ∗ ,<br />
and the approximation is updated, i.e.,<br />
u (k+1) = u (k)<br />
N−n + v (k) .<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 42
The Faul-Powell Algorithm<br />
Step 3<br />
Finally, the residuals are updated, i.e.,<br />
r (k+1)<br />
i<br />
= f (x i ) − u (k+1) (x i ), i = 1, . . . , N. (10)<br />
Remark<br />
The outer iteration (on k) is now repeated unless the largest <strong>of</strong> these<br />
residuals is small enough.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 43
The Faul-Powell Algorithm<br />
Algorithm (Pre-processing step)<br />
Choose n<br />
For 1 ≤ j ≤ N − n do<br />
Determine the index set L j<br />
Find the coefficients b jl <strong>of</strong> the approximate cardinal function Ψ j by<br />
solving<br />
Ψ j (x i ) = δ jk , i ∈ L j<br />
end<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 44
The Faul-Powell Algorithm<br />
Algorithm (Faul-Powell)<br />
Input data locations X , associated values <strong>of</strong> f , tolerance tol > 0<br />
Perform pre-processing step<br />
Initialize: k = 0, u (k)<br />
0<br />
= 0, r (k)<br />
i<br />
= f (x i ), i = 1, . . . , N, e = max<br />
i=1,...,N<br />
While e > tol do<br />
Update<br />
u (k)<br />
j<br />
|r<br />
(k)<br />
i<br />
|<br />
= u (k)<br />
j−1 + Ψ ∑ (<br />
)<br />
j<br />
b jl f (x l ) − u (k)<br />
j−1<br />
b (x l) , 1 ≤ j ≤ N − n<br />
jj<br />
l∈L j<br />
Solve the interpolation problem<br />
v (k) (x i ) = f (x i ) − u (k)<br />
N−n (x i),<br />
Update the approximation<br />
Compute new residuals r (k+1)<br />
i<br />
Set new value for e =<br />
Increment k = k + 1<br />
u (k+1)<br />
0<br />
= u (k)<br />
N−n + v (k)<br />
max<br />
i=1,...,N<br />
i ∈ L ∗<br />
= f (x i ) − u (k+1)<br />
0<br />
(x i ), i = 1, . . . , N<br />
(k+1)<br />
|r<br />
i<br />
|<br />
end<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 45
The Faul-Powell Algorithm<br />
Remark<br />
Faul and Powell prove that this algorithm converges to the solution<br />
<strong>of</strong> the original interpolation problem.<br />
One needs to make sure that the residuals are evaluated<br />
efficiently by using, e.g.,<br />
a fast multipole expansion,<br />
fast Fourier transform, or<br />
compactly supported kernels.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 46
The Faul-Powell Algorithm<br />
Remark<br />
In its most basic form the Krylov subspace algorithm <strong>of</strong> Faul and<br />
Powell can also be explained as a dual approach to the greedy<br />
residual iteration algorithm <strong>of</strong> Schaback and Wendland.<br />
Instead <strong>of</strong> defining appropriate sets <strong>of</strong> points Y k , in the Faul and<br />
Powell algorithm one picks certain subspaces U k <strong>of</strong> the native<br />
space.<br />
In particular, if U k is the one-dimensional space U k = span{Ψ k }<br />
(where Ψ k is a local approximation to the cardinal function) we get<br />
the Schaback-Wendland algorithm described above.<br />
For more details see [Schaback and Wendland (2000b)].<br />
Implementation <strong>of</strong> this algorithm is omitted.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 47
References I<br />
Appendix<br />
References<br />
Berlinet, A., Thomas-Agnan, C. (2004).<br />
Reproducing Kernel Hilbert Spaces in Probability and Statistics.<br />
Kluwer, Dordrecht.<br />
Buhmann, M. D. (2003).<br />
Radial Basis Functions: Theory and Implementations.<br />
Cambridge University Press.<br />
Fasshauer, G. E. (2007).<br />
Meshfree Approximation Methods with MATLAB.<br />
World Scientific Publishers.<br />
Higham, D. J. and Higham, N. J. (2005).<br />
MATLAB Guide.<br />
SIAM (2nd ed.), Philadelphia.<br />
Iske, A. (2004).<br />
Multiresolution Methods in Scattered Data Modelling.<br />
Lecture Notes in Computational Science and Engineering 37, Springer Verlag<br />
(Berlin).<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 48
References II<br />
Appendix<br />
References<br />
G. Wahba (1990).<br />
Spline Models for Observational Data.<br />
CBMS-NSF Regional Conference Series in <strong>Applied</strong> <strong>Mathematics</strong> 59, SIAM<br />
(Philadelphia).<br />
Wendland, H. (2005a).<br />
Scattered Data Approximation.<br />
Cambridge University Press (Cambridge).<br />
Faul, A. C. and Powell, M. J. D. (1999).<br />
Pro<strong>of</strong> <strong>of</strong> convergence <strong>of</strong> an iterative technique for thin plate spline interpolation in<br />
two dimensions.<br />
Adv. Comput. Math. 11, pp. 183–192.<br />
Faul, A. C. and Powell, M. J. D. (2000).<br />
Krylov subspace methods for radial basis function interpolation.<br />
in Numerical Analysis 1999 (Dundee), Chapman & Hall/CRC (Boca Raton, FL),<br />
pp. 115–141.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 49
References III<br />
Appendix<br />
References<br />
Girosi, F. (1998).<br />
An equivalence between sparse approximation and support vector machines.<br />
Neural Computation 10, pp. 1455–1480.<br />
Schaback, R. and Wendland, H. (2000a).<br />
Numerical techniques based on radial basis functions.<br />
in Curve and Surface Fitting: Saint-Malo 1999, A. Cohen, C. Rabut, and<br />
L. L. Schumaker (eds.), Vanderbilt University Press (Nashville, TN), 359-374.<br />
Schaback, R. and Wendland, H. (2000b).<br />
Adaptive greedy techniques for approximate solution <strong>of</strong> large RBF systems.<br />
Numer. Algorithms 24, pp. 239–254.<br />
Temlyakov, V. N. (1998).<br />
The best m-term approximation and greedy algorithms.<br />
Adv. in Comp. Math. 8, pp. 249–265.<br />
fasshauer@iit.edu MATH 590 – <strong>Chapter</strong> <strong>33</strong> 50