Contents

AMATH 581 ( c○J. N. Kutz) 9is O(∆t 4 ). The key to this method, as well as any of the other Runge-Kuttaschemes, is the use of intermediate time-steps to improve accuracy. For the4th order scheme presented here, a graphical representation ofthisderivativesampling at intermediate time-steps is shown in Fig. 2.Adams method: multi-stepping techniquesThe development of the Runge-Kutta schemes rely on the definition of thederivative and Taylor expansions. Another approach to solving (1.1.1) is to startwith the fundamental theorem of calculus [2]. Thus the differential equation canbe integrated over a time-step ∆t to give∫dyt+∆tdt = f(y,t) ⇒ y(t +∆t) − y(t) = f(t, y)dt . (1.1.16)And once again using our iteration notation we findt∫ tn+1y n+1 = y n + f(t, y)dt . (1.1.17)t nThis iteration relation is simply a restatement of (1.1.7) with ∆t·φ = ∫ t n+1t nf(t, y)dt.However, at this point, no approximations have been made and (1.1.17) is exact.The numerical solution will be found by approximating f(t, y) ≈ p(t, y) wherep(t, y) isapolynomial.Thustheiterationschemeinthisinstancewill be givenby∫ tn+1y n+1 ≈ y n + p(t, y)dt . (1.1.18)t nIt only remains to determine the form of the polynomial to be used in theapproximation.The Adams-Bashforth suite of computational methods uses the current pointand a determined number of past points to evaluate the future solution. Aswith the Runge-Kutta schemes, the order of accuracy is determined by thechoice of φ. In the Adams-Bashforth case, this relates directly to the choiceof the polynomial approximation p(t, y). A first-order scheme can easily beconstructed by allowingp 1 (t) =constant=f(t n , y n ) , (1.1.19)where the present point and no past points are used to determine the value ofthe polynomial. Inserting this first-order approximation into (1.1.18) results inthe previously found Euler schemey n+1 = y n +∆t · f(t n , y n ) . (1.1.20)

AMATH 581 ( c○J. N. Kutz) 10Alternatively, we could assume that the polynomial used both thecurrentpointand the previous point so that a second-order scheme resulted. The linearpolynomial which passes through these two points is given byp 2 (t) =f n−1 + f n − f n−1(t − t n−1 ) . (1.1.21)∆tWhen inserted into (1.1.18), this linear polynomial yields∫ tn+1(y n+1 = y n + f n−1 + f )n − f n−1(t − t n ) dt . (1.1.22)t n∆tUpon integration and evaluation at the upper and lower limits, we find thefollowing 2nd order Adams-Bashforth schemey n+1 = y n + ∆t2 [3f(t n, y n ) − f(t n−1 , y n−1 )] . (1.1.23)In contrast to the Runge-Kutta method, this is a two-step algorithm which requirestwo initial conditions. This technique can be easily generalized to includemore past points and thus higher accuracy. However, as accuracy is increased,so are the number of initial conditions required to step forward one time-step∆t. Asidefromthefirst-orderaccuratescheme,anyimplementation of Adams-Bashforth will require a boot strap to generate a second “initial condition” forthe solution iteration process.The Adams-Bashforth scheme uses current and past points to approximatethe polymial p(t, y) in(1.1.18). Ifinsteadafuturepoint,thepresent,andthepast is used, then the scheme is known as an Adams-Moulton method. Asbefore,afirst-orderschemecaneasilybeconstructedbyallowingp 1 (t) =constant=f(t n+1 , y n+1 ) , (1.1.24)where the future point and no past and present points are used to determine thevalue of the polynomial. Inserting this first-order approximation into (1.1.18)results in the backward Euler schemey n+1 = y n +∆t · f(t n+1 , y n+1 ) . (1.1.25)Alternatively, we could assume that the polynomial used both thefuturepointand the current point so that a second-order scheme resulted. Thelinearpolynomialwhich passes through these two points is given byp 2 (t) =f n + f n+1 − f n(t − t n ) . (1.1.26)∆tInserted into (1.1.18), this linear polynomial yields∫ tn+1(y n+1 = y n + f n + f )n+1 − f n(t − t n ) dt . (1.1.27)t n∆t

AMATH 581 ( c○J. N. Kutz) 11Upon integration and evaluation at the upper and lower limits, we find thefollowing 2nd order Adams-Moulton schemey n+1 = y n + ∆t2 [f(t n+1, y n+1 )+f(t n , y n )] . (1.1.28)Once again this is a two-step algorithm. However, it is categorically differentthan the Adams-Bashforth methods since it results in an implicit scheme,i.e. the unknown value y n+1 is specified through a nonlinear equation (1.1.28).The solution of this nonlinear system can be very difficult, thus making explicitschemes such as Runge-Kutta and Adams-Bashforth, which are simpleiterations, more easily handled. However, implicit schemes canhaveadvantageswhen considering stability issues related to time-stepping. This is explored furtherin the notes.One way to circumvent the difficulties of the implicit stepping methodwhilestill making use of its power is to use a Predictor-Corrector method. Thisschemedraws on the power of both the Adams-Bashforth and Adams-Moulton schemes.In particular, the second order implicit scheme given by (1.1.28) requires thevalue of f(t n+1 , y n+1 )intherighthandside. Ifwecanpredict(approximate)this value, then we can use this predicted value to solve (1.1.28) explicitly. Thuswe begin with a predictor step to estimate y n+1 so that f(t n+1 , y n+1 )canbeevaluated. We then insert this value into the right hand side of (1.1.28) andexplicitly find the corrected value of y n+1 .Thesecond-orderpredictor-correctorsteps are then as follows:predictor (Adams-Bashforth): y P n+1 =y n + ∆t2 [3f n − f n−1 ] (1.1.29a)corrector (Adams-Moulton): y n+1 =y n + ∆t2 [f(t n+1, y P n+1 )+f(t n, y n )]. (1.1.29b)Thus the scheme utilizes both explicit and implicit time-stepping schemes withouthaving to solve a system of nonlinear equations.Higher order differential equationsThus far, we have considered systems of first order equations. Higherorderdifferentialequations can be put into this form and the methods outlined here canbe applied. For example, consider the third-order, nonhomogeneous, differentialequationd 3 u du+ u2 +cost · u = g(t) . (1.1.30)dt3 dtBy definingy 1 = uy 2 = dudt(1.1.31a)(1.1.31b)

AMATH 581 ( c○J. N. Kutz) 12y 3 = d2 udt 2 ,(1.1.31c)we find that dy 3 /dt = d 3 u/dt 3 .definitions of y i we find thatUsing the original equation along with thedy 1dt = y 2(1.1.32a)dy 2dt = y 3(1.1.32b)dy 3dt = d3 u dudt 3 = −u2dt − cos t · u + g(t) =−y2 1 y 2 − cos t · y 1 + g(t)(1.1.32c)which results in the original differential equation (1.1.1) considered previously⎛dydt = d ⎝ y ⎞ ⎛⎞1y 2y 2⎠ = ⎝y 3⎠ = f(y,t) . (1.1.33)dty 3 −y1 2y 2 − cos t · y 1 + g(t)At this point, all the time-stepping techniques developed thus far can be appliedto the problem. It is imperative to write any differential equation as a first-ordersystem before solving it numerically with the time-stepping schemesdevelopedhere.MATLAB commandsThe time-stepping schemes considered here are all available intheMATLABsuite of differential equation solvers. The following are a few of the most commonsolvers:• ode23: second-order Runge-Kutta routine• ode45: fourth-order Runge-Kutta routine• ode113: variable order predictor-corrector routine• ode15s: variable order Gear method for stiff problems [3, 4]1.2 Error analysis for time-stepping routinesAccuracy and stability are fundamental to numerical analysis and are the key factorsin evaluating any numerical integration technique. Therefore, it is essentialto evaluate the accuracy and stability of the time-stepping schemes developed.Rarely does it occur that both accuracy and stability work in concert. In fact,they often are offsetting and work directly against each other. Thus a highlyaccurate scheme may compromise stability, whereas a low accuracy scheme mayhave excellent stability properties.

AMATH 581 ( c○J. N. Kutz) 13We begin by exploring accuracy. In the context of time-stepping schemes,the natural place to begin is with Taylor expansions. Thus we consider theexpansiony(t +∆t) =y(t)+∆t · dy(t) + ∆t2dt 2 · d2 y(c)dt 2 (1.2.1)where c ∈ [t, t +∆t]. Since we are considering dbfy/dt = f(t, y), the aboveformula reduces to the Euler iteration schemey n+1 = y n +∆t · f(t n , y n )+O(∆t 2 ) . (1.2.2)It is clear from this that the truncation error is O(∆t 2 ). Specifically, the truncationerror is given by ∆t 2 /2 · d 2 y(c)/dt 2 .Of importance is how this truncation error contributes to the overallerrorin the numerical solution. Two types of error are important to identify: localand global error. Each is significant in its own right. However, in practice weare only concerned with the global (cumulative) error. The global discretizationerror is given byE k = y(t k ) − y k (1.2.3)where y(t k )istheexactsolutionandy k is the numerical solution. The localdiscretization error is given byɛ k+1 = y(t k+1 ) − (y(t k )+∆t · φ) (1.2.4)where y(t k+1 )istheexactsolutionandy(t k )+∆t·φ is a one-step approximationover the time interval t ∈ [t n ,t n+1 ].For the Euler method, we can calculate both the local and global error.Given a time-step ∆t and a specified time interval t ∈ [a, b], we have after Ksteps that ∆t · K = b − a. Thuswefindlocal: ɛ k = ∆t22global:K∑ ∆t 2E k =2j=1= ∆t22d 2 y(c k )dt 2 ∼ O(∆t 2 ) (1.2.5a)d 2 y(c)dt 2d 2 y(c j )dt 2≈ ∆t22d 2 y(c)dt 2· K· b − a = b − a∆t 2 ∆t · d2 y(c)dt 2 ∼ O(∆t) (1.2.5b)which gives a local error for the Euler scheme which is O(∆t 2 )andaglobalerror which is O(∆t). Thus the cumulative error is large for the Euler scheme,i.e. it is not very accurate.Asimilarprocedurecanbecarriedoutforalltheschemesdiscussed thusfar, including the multi-step Adams schemes. Table 1 illustrates various schemesand their associated local and global errors. The error analysis suggests that theerror will always decrease in some power of ∆t. Thusitistemptingtoconcludethat higher accuracy is easily achieved by taking smaller time steps ∆t. Thiswould be true if not for round-off error in the computer.

AMATH 581 ( c○J. N. Kutz) 14scheme local error ɛ k global error E kEuler O(∆t 2 ) O(∆t)2nd order Runge-Kutta O(∆t 3 ) O(∆t 2 )4th order Runge-Kutta O(∆t 5 ) O(∆t 4 )2nd order Adams-Bashforth O(∆t 3 ) O(∆t 2 )Table 1: Local and global discretization errors associated with various timesteppingschemes.Round-off and step-sizeAn unavoidable consequence of working with numerical computations is roundofferror. When working with most computations, double precision numbersare used. This allows for 16-digit accuracy in the representation of a givennumber. This round-off has significant impact upon numerical computationsand the issue of time-stepping.As an example of the impact of round-off, we consider the Euler approximationto the derivativedydt ≈ y n+1 − y n+ ɛ(y n , ∆t) (1.2.6)∆twhere ɛ(y n , ∆t) measuresthetruncationerror.Uponevaluatingthisexpressionin the computer, round-off error occurs so thaty n+1 = Y n+1 + e n+1 . (1.2.7)Thus the combined error between the round-off and truncation gives the followingexpression for the derivative:dydt = Y n+1 − Y n+ E n (y n , ∆t) (1.2.8)∆twhere the total error, E n ,isthecombinationofround-offandtruncationsuchthatE n = E round + E trunc = e n+1 − e n− ∆t2 d 2 y(c)∆t 2 dt 2 . (1.2.9)We now determine the maximum size of the error. In particular,wecanboundthe maximum value of round-off and the derivate to be|e n+1 |≤e r (1.2.10a)|−e n |≤e rM =maxc∈[t n,t n+1](1.2.10b){∣ ∣ ∣∣∣ d 2 }y(c) ∣∣∣dt 2 . (1.2.10c)

AMATH 581 ( c○J. N. Kutz) 15This then gives the maximum error to be|E n |≤ e r + e r∆t+ ∆t22 M = 2e r∆t + ∆t2 M. (1.2.11)2To minimize the error, we require that ∂|E n |/∂(∆t) = 0. Calculating thisderivative gives∂|E n |∂(∆t) = − 2e r+ M∆t =0, (1.2.12)∆t2 so that∆t =( ) 1/3 2er. (1.2.13)MThis gives the step-size resulting in a minimum error. Thus the smallest stepsizeis not necessarily the most accurate. Rather, a balance between round-offerror and truncation error is achieved to obtain the optimal step-size.StabilityThe accuracy of any scheme is certainly important. However, it is meaninglessif the scheme is not stable numerically. The essense of a stable scheme: thenumerical solutions do not blow up to infinity. As an example, consider thesimple differential equationdy= λy (1.2.14)dtwithy(0) = y 0 . (1.2.15)The analytic solution is easily calculated to be y(t) =y 0 exp(λt). However, ifwe solve this problem numerically with a forward Euler method wefindy n+1 = y n +∆t · λy n =(1+λ∆t)y n . (1.2.16)After N steps, we find this iteration scheme yieldsy N =(1+λ∆t) N y 0 . (1.2.17)Given that we have a certain amount of round off error, the numerical solutionwould then be given byy N =(1+λ∆t) N (y 0 + e) . (1.2.18)The error then associated with this scheme is given byE =(1+λ∆t) N e. (1.2.19)

AMATH 581 ( c○J. N. Kutz) 16At this point, the following observations can be made. For λ>0, the solutiony N →∞in Eq. (1.2.18) as N →∞. So although the error also grows, it maynot be significant in comparison to the size of the numerical solution.In contrast, Eq. (1.2.18) for λ < 0ismarkedlydifferent. Forthiscase,y N → 0inEq.(1.2.18)asN → ∞. The error, however, can dominate inthis case. In particular, we have the two following cases for the error given by(1.2.19):I: |1+λ∆t| < 1 then E → 0 (1.2.20a)II: |1+λ∆t| > 1 then E →∞. (1.2.20b)In case I, the scheme would be considered stable. However, case II holds and isunstable provided ∆t >−2/λ.Ageneraltheoryofstabilitycanbedevelopedforanyone-step time-steppingscheme. Consider the one-step recursion relation for an M × M systemAfter N steps, the algorithm yields the solutiony n+1 = Ay n . (1.2.21)y N = A N y 0 , (1.2.22)where y 0 is the initial vector. A well known result from linear algebra isthatA N = S −1 Λ N S (1.2.23)where S is the matrix whose columns are the eigenvectors of A, and⎛⎞⎛⎞λ 1 0 ··· 0λ N 1 0 ··· 00 λ 2 0 ···Λ = ⎜⎝.. .. .⎟⎠ → 0 λ NΛN 2 0 ···= ⎜⎝.. .. .⎟⎠ (1.2.24)0 ··· 0 λ M 0 ··· 0 λ N Mis a diagnonal matrix whose entries are the eigenvalues of A. ThusuponcalculatingΛ N ,weareonlyconcernedwiththeeigenvalues.Inparticular,instabilityoccurs if R{λ i } > 1fori =1, 2,...,M. This method can be easily generalizedto two-step schemes (Adams methods) by considering y n+1 = Ay n + By n−1 .Lending further significance to this stability analysis is its connection withpractical implementation. We contrast the difference in stability between theforward and backward Euler schemes. The forward Euler scheme hasalreadybeen considered in (1.2.16)-(1.2.19). The backward Euler displays significantdifferences in stability. If we again consider (1.2.14) with (1.2.15), the backwardEuler method gives the iteration schemey n+1 = y n +∆t · λy n+1 , (1.2.25)

AMATH 581 ( c○J. N. Kutz) 17Figure 3: Regions for stable stepping for the forward Euler and backward Eulerschemes.which after N steps leads to( ) N1y N =y 0 . (1.2.26)1 − λ∆tThe round-off error associated with this scheme is given by( ) N1E =e. (1.2.27)1 − λ∆tBy letting z = λ∆t be a complex number, we find the following criteria to yieldunstable behavior based upon (1.2.19) and (1.2.27)forward Euler: |1+z| > 1 (1.2.28a)backward Euler:1∣1 − z ∣ > 1 .(1.2.28b)Figure 3 shows the regions of stable and unstable behavior as afunctionofz.It is observed that the forward Euler scheme has a very small range of stabilitywhereas the backward Euler scheme has a large range of stability. This largestability region is part of what makes implicit methods so attractive. Thusstability regions can be calculated. However, control of the accuracyisalsoessential.1.3 Boundary value problems: the shooting methodTo this point, we have only considered the solutions of differential equations forwhich the initial conditions are known. However, many physical applicationsdo not have specified initial conditions, but rather some given boundary (constraint)conditions. A simple example of such a problem is the second-orderboundary value problemd 2 (ydt 2 = f t, y, dy )(1.3.1)dt

AMATH 581 ( c○J. N. Kutz) 18y(t)y’’=f(t,y,y’)y(b)ay(a)btFigure 4: Graphical depiction of the structure of a typical solution to a boundaryvalue problem with constraints at t = a and t = b.on t ∈ [a, b] withthegeneralboundaryconditionsdy(a)α 1 y(a)+β 1dtdy(b)α 2 y(b)+β 2dt= γ 1 (1.3.2a)= γ 2 . (1.3.2b)Thus the solution is defined over a specific interval and must satisfy the relations(1.3.2) at the end points of the interval. Figure 4 gives a graphical representationof a generic boundary value problem solution. We discuss thealgorithmnecessary to make use of the time-stepping schemes in order to solvesuchaproblem.The Shooting MethodThe boundary value problems constructed here require information at the presenttime (t = a) andafuturetime(t = b). However, the time-stepping schemes developedpreviously only require information about the starting time t = a. Someeffort is then needed to reconcile the time-stepping schemes with the boundaryvalue problems presented here.We begin by reconsidering the generic boundary value problemd 2 (ydt 2 = f t, y, dy )(1.3.3)dton t ∈ [a, b] withtheboundaryconditionsy(a) =αy(b) =β.(1.3.4a)(1.3.4b)

AMATH 581 ( c○J. N. Kutz) 19y’’=f(t,y,y’)y(t)y(b) >ββA=A2targety(b) =βαA=Ay(a)1y’’=f(t,y,y’)y(b)

AMATH 581 ( c○J. N. Kutz) 20y(t)A=A2y’’=f(t,y,y’)βαA=A 1y(a)A=A 35A=A4A=Ay(b) =βtargetA

AMATH 581 ( c○J. N. Kutz) 21n(x)/n 010.80.60.40.20−3 −2 −1 0 1 2 3xFigure 7: Plot of the spatial function n(x).Eigenvalues and Eigenfunctions: The Infinite DomainBoundary value problems often arise as eigenvalue systems for which the eigenvalueand eigenfunction must both be determined. As an example of such aproblem, we consider the second order differential equation on the infinite lined 2 ψ ndx 2 +[n(x) − β n] ψ n =0 (1.3.6)with the boundary conditions ψ n (x) → 0asx →±∞. For this example, weconsider the spatial function n(x) whichisgivenby{1 −|x|20 ≤|x| ≤1n(x) =n 00 |x| > 1(1.3.7)with n 0 being an arbitrary constant. Figure 7 shows the spatial dependence ofn(x). The parameter β n in this problem is the eigenvalue. For each eigenvalue,we can calculate a normalized eigenfunction ψ n . The standard normalizationrequires ∫ ∞−∞ |ψ n| 2 dx =1.Although the boundary conditions are imposed as x →±∞,computationallywe require a finite domain. We thus define our computational domain to bex ∈ [−L, L] whereL ≫ 1. Since n(x) =0for|x| > 1, the governing equationreduces tod 2 ψ ndx 2 − β nψ n =0 |x| > 1 (1.3.8)which has the general solutionψ n = c 1 exp( √ β n x)+c 2 exp(− √ β n x) (1.3.9)

AMATH 581 ( c○J. N. Kutz) 22ψ(x)1.510.52β 1=86.79 β 2=67.1410−10−2 −1 0 1 22−2−2 −1 0 1 22ψ(x)10β 3=47.51 β 4=28.0610−1−1−2 −1 0 1 2x−2−2 −1 0 1 2xFigure 8: Plot of the first four eigenfunctions along with their eigenvalues β n .For this example, L =2andn 0 =100. Theseeigenmodestructuresaretypicalof those found in quantum mechanics and electromagnetic waveguides.for β n ≥ 0. Note that we can only consider values of β n ≥ 0sinceforβ n < 0,the general solution becomes ψ n = c 1 cos( √ |β n |x) +c 2 sin(− √ |β n |x) whichdoes not decay to zero as x →±∞.Inordertoensurethatthedecayboundaryconditions are satisfied, we must eliminate one of the two linearly independentsolutions of the general solution. In particular, we must havex →∞: ψ n = c 2 exp(− √ β n x) (1.3.10a)x →−∞: ψ n = c 1 exp( √ β n x) . (1.3.10b)Thus the requirement that the solution decays at infinity eliminates one of thetwo linearly independent solutions. Alternatively, we could think of this situationas being a case where only one linearly independent solution is allowed asx →±∞.Butasinglelinearlyindependentsolutioncorrespondstoafirstorderdifferential equation. Therefore, the decay solutions (1.3.10) can equivalently

AMATH 581 ( c○J. N. Kutz) 23be thought of as solutions to the following first order equations:x →∞:x →−∞:dψ ndx + √ β n ψ n =0dψ ndx − √ β n ψ n =0.(1.3.11a)(1.3.11b)From a computational viewpoint then, the effective boundary conditions to beconsidered on the computational domain x ∈ [−L, L] arethefollowingx = L :x = −L :dψ n (L)= − √ β n ψ n (L)dx(1.3.12a)dψ n (−L)= √ β n ψ n (−L) .dx(1.3.12b)In order to solve the problem, we write the governing differential equationas a system of equations. Thus we let x 1 = ψ n and x 2 = dψ n /dx which givesx ′ 1 = ψ′ n = x 2x ′ 2 = ψ′′ n =[β n − n(x)] ψ n =[β n − n(x)] x 1 .(1.3.13a)(1.3.13b)In matrix form, we can write the governing system as()x ′ 0 1=x (1.3.14)β n − n(x) 0where x =(x 1 x 2 ) T =(ψ n dψ n /dx) T .Theboundaryconditions(1.3.12)arex = L : x 2 = − √ β n x 1 (1.3.15a)x = −L : x 2 = √ β n x 1 . (1.3.15b)The formulation of the boundary value problem is thus complete. It remains todevelop an algorithm to find the eigenvalues β n and corresponding eigenfunctionsψ n . Figure 8 illustrates the first four eigenfunctions and theirassociateeigenvalues for n 0 =100andL =2.1.4 Implementation of shooting and convergence studiesThe implementation of the shooting scheme relies on the effective use of a timesteppingalgorithm along with a root finding method for choosing the appropriateinitial conditions which solve the boundary value problem. The specificsystem to be considered is similar to that developed in the last lecture. Weconsider(x ′ =0 1β n − n(x) 0)x (1.4.1)

AMATH 581 ( c○J. N. Kutz) 24where x =(x 1 x 2 ) T =(ψ n dψ n /dx) T .Theboundaryconditionsaresimplifiedin this case to bex =1: ψ n (1) = x 1 (1) = 0 (1.4.2a)x = −1 : ψ n (−1) = x 1 (−1) = 0 . (1.4.2b)At this stage, we will also assume that n(x) =n 0 for simplicity.With the problem thus defined, we turn our attention to the key aspects inthe computational implementation of the boundary value problem solver. Theseare• FOR loops• IF statements• time-stepping algorithms: ode23, ode45, ode113, ode15s• step-size control• code development and flowEvery code will be controlled by a set of FOR loops and IF statements. Itis imperative to have proper placement of these control statements in order forthe code to operate successfully.ConvergenceIn addition to developing a successful code, it is reasonable toaskwhetheryournumerical solution is actually correct. Thus far, the premise has been thatdiscretization should provide an accurate approximation to thetruesolutionprovided the time-step ∆t is small enough. Although in general this philosophyis correct, every numerical algorithm should be carefully checked to determineif it indeed converges to the true solution. The time-stepping schemes consideredpreviously already hint at how the solutions should converge: fourth-orderRunge-Kutta converges like ∆t 4 ,second-orderRunge-Kuttaconvergeslike∆t 2 ,and second-order predictor-corrector schemes converge like ∆t 2 .Thusthealgorithmfor checking convergence is as follows:1. Solve the differential equation using a time-step ∆t∗ which is very small.This solution will be considered the exact solution. Recall that we wouldin general like to take ∆t as large as possible for efficiency purposes.2. Using a much larger time-step ∆t, solvethedifferentialequationandcomparethe numerical solution with that generated from ∆t∗. Cutthistimestepin half and compare once again. In fact, continue cutting thetimestepin half: ∆t, ∆t/2, ∆t/4, ∆t/8, ··· in order to compare the differencein the exact solution to this hierarchy of solutions.

AMATH 581 ( c○J. N. Kutz) 253. The difference between any run ∆t∗ and ∆t is considered the error. Althoughthere are many definitions of error, a practial error measurement[is the root mean-square error E = (1/N ) ∑ ]N1/2.i=1 |y ∆t∗ − y ∆t | 2 Oncecalculated, it is possible to verify the convergence law of ∆t 2 ,forinstance,with a second-order Runge-Kutta.Flow ControlIn order to begin coding, it is always prudent to construct the basicstructureof the algorithm. In particular, it is good to determine the number of FORloops and IF statements which may be required for the computations. Whatis especially important is determining the hierarchy structure for the loops. Tosolve the boundary value problem proposed here, we require two FOR loopsand one IF statement block. The outermost FOR loop of the code shoulddetermine the number of eigenvalues and eigenmodes to be searched for. Withinthis FOR loop exists a second FOR loop which iterates the shooting methodso that the solution converges to the correct boundary value solution. Thissecond FOR loop has a logical IF statement which needs to check whetherthe solution has indeed converged to the boundary value solution, or whetheradjustment of the value of β n is necessary and the iteration procedure needsto be continued. Figure 9 illustrates the backbone of the numerical code forsolving the boundary value problem. It includes the two FOR loops and logicalIF statement block as the core of its algorithmic structure. For a nonlinearproblem, a third FOR loop would be required for A in order to achieve thenormalization of the eigenfunctions to unity.The various pieces of the code are constructed here using the MATLAB programminglanguage. We begin with the initialization of the parameters.Initializationclear all;close all;% clear all previously defined variables% clear all previously defined figurestol=10^(-4); % define a tolerance level to be achieved% by the shooting algorithmcol=[’r’,’b’,’g’,’c’,’m’,’k’]; % eigenfunction colorsn0=100; % define the parameter n0A=1; % define the initial slope at x=-1x0=[0 A]; % initial conditions: x1(-1)=0, x1’(-1)=Axp=[-1 1]; % define the span of the computational domainUpon completion of the initialization process for the parameters which are notinvolved in the main loop of the code, we move into the main FOR loop which

AMATH 581 ( c○J. N. Kutz) 26initializeparameterschoose Achooseβ nsolve ODEsmode loopIF statement:root solve forβ nβnloopIF converged:plot and save dataIF not converged:new value ofβ nnormalize andchoose new modeFigure 9: Basic algorithm structure for solving the boundary valueproblem.Two FOR loops are required to step through the values of β n and A along withasingleIFstatementblocktocheckforconvergenceofthesolutionsearches out a specified number of eigenmodes. Embedded in this FOR loopis a second FOR loop which attemps different values of β n until the correcteigenvalue is found. An IF statement is used to check the convergence of valuesof β n to the appropriate value.Main Programbeta_start=n0; % beginning value of betafor modes=1:5 % begin mode loopbeta=beta_start; % initial value of eigenvalue betadbeta=n0/100; % default step size in betafor j=1:1000 % begin convergence loop for beta[t,y]=ode45(’shoot2’,xp,x0,[],n0,beta); % solve ODEsif abs(y(end,1)-0) < tol % check for convergencebeta% write out eigenvalue

AMATH 581 ( c○J. N. Kutz) 27break% get out of convergence loopendif (-1)^(modes+1)*y(end,1)>0 % this IF statement blockbeta=beta-dbeta; % checks to see if betaelse % needs to be higher or lowerbeta=beta+dbeta/2; % and uses bisection todbeta=dbeta/2; % converge to the solutionend %end % end convergence loopbeta_start=beta-0.1; % after finding eigenvalue, pick% new starting value for next modenorm=trapz(t,y(:,1).*y(:,1)) % calculate the normalizationplot(t,y(:,1)/sqrt(norm),col(modes)); hold on % plot modesend % end mode loopThe code uses ode45, whichisafourth-orderRunge-Kuttamethod,tosolvethe differential equation and advance the solution. The function shoot2.m iscalled in this routine. For the differential equation considered here, the functionshoot2.m would be the following:shoot2.mfunction rhs=shoot2(xspan,x,dummy,n0,beta)rhs=[ x(2)(beta-n0)*x(1) ];This code will find the first five eigenvalues and plot their correspondingnormalized eigenfunctions. The bisection method implemented to adjust thevalues of β n to find the boundary value solution is based upon observationsofthestructure of the even and odd eigenmodes. In general, it is always a good idea tofirst explore the behavior of the solutions of the boundary value problem beforewriting the shooting routine. This will give important insights into the behaviorof the solutions and will allow for a proper construction of an accurateandefficient bisection method. Figure 10 illustrates several characteristic featuresof this boundary value problem. In Fig. 10(a) and 10(b), the behavior of thesolution near the first even and first odd solution is exhibited. From Fig. 10(a)it is seen that for the even modes increasing values of β bring the solutionfrom ψ n (1) > 0toψ n (1) < 0. In contrast, odd modes go from ψ n (1) < 0toψ n (1) > 0asβ is increased. This observation forms the basis for the bisectionmethod developed in the code. Figure 10(c) illustrates the first four normalizedeigenmodes along with their corresponding eigenvalues.

AMATH 581 ( c○J. N. Kutz) 281ψ(x)0.5−0.5ψ(x)021β=98β=95(a)−1 −0.5 0 0.5 1x10.50−0.5−1(c)β=95β=85(b)−1 −0.5 0 0.5 1x0−1−2β 4=60.53β 3=77.80β 2=90.13β 1=97.53−1 −0.5 0 0.5 1xFigure 10: In (a) and (b) the behavior of the solution near the first even andfirst odd solution is depicted. Note that for the even modes increasing values ofβ bring the solution from ψ n (1) > 0toψ n (1) < 0. In contrast, odd modes gofrom ψ n (1) < 0toψ n (1) > 0asβ is increased. In (c) the first four normalizedeigenmodes along with their corresponding eigenvalues are illustrated for n 0 =100.1.5 Boundary value problems: direct solve and relaxationThe shooting method is not the only method for solving boundary value problems.The direct method of solution relies on Taylor expanding the differentialequation itself. For linear problems, this results in a matrix problem of theform Ax = b. Fornonlinearproblems,anonlinearsystemofequationsmust besolved using a relaxation scheme, i.e. a Newton or Secant method. The prototypicalexample of such a problem is the second-order boundary value problemd 2 (ydt 2 = f t, y, dy )(1.5.1)dt

AMATH 581 ( c○J. N. Kutz) 29O(∆t 2 )center-differenceschemesf ′ (t) =[f(t +∆t) − f(t − ∆t)]/2∆tf ′′ (t) =[f(t +∆t) − 2f(t)+f(t − ∆t)]/∆t 2f ′′′ (t) =[f(t +2∆t) − 2f(t +∆t)+2f(t − ∆t) − f(t − 2∆t)]/2∆t 3f ′′′′ (t) =[f(t +2∆t) − 4f(t +∆t)+6f(t) − 4f(t − ∆t)+f(t − 2∆t)]/∆t 4Table 2: Second-order accurate center-difference formulas.on t ∈ [a, b] withthegeneralboundaryconditionsdy(a)α 1 y(a)+β 1dtdy(b)α 2 y(b)+β 2dt= γ 1 (1.5.2a)= γ 2 . (1.5.2b)Thus the solution is defined over a specific interval and must satisfy the relations(1.5.2) at the end points of the interval.Before considering the general case, we simplify the method by consideringthe linear boundary value problemd 2 ydt 2on t ∈ [a, b] withthesimplifiedboundaryconditions= p(t)dy + q(t)y + r(t) (1.5.3)dty(a) =αy(b) =β.(1.5.4a)(1.5.4b)Taylor expanding the differential equation and boundary conditions will generatethe linear system of equations which solve the boundary value problem.To see how the Taylor expansions are useful, consider the following twoTaylor series:df (t)f(t +∆t) =f(t)+∆tdtf(t − ∆t) =f(t) − ∆tdf (t)dt+ ∆t22!+ ∆t22!d 2 f(t)dt 2d 2 f(t)dt 2where c i ∈ [a, b]. Subtracting these two expressions givesdf (t)f(t +∆t) − f(t − ∆t) =2∆t + ∆t3dt 3!+ ∆t33!− ∆t33!d 3 f(c 1 )dt 3d 3 f(c 2 )dt 3(1.5.5a)(1.5.5b)( d 3 f(c 1 )dt 3 + d3 f(c 2 )dt 3 ). (1.5.6)

AMATH 581 ( c○J. N. Kutz) 30O(∆t 4 )center-differenceschemesf ′ (t) =[−f(t +2∆t)+8f(t +∆t) − 8f(t − ∆t)+f(t − 2∆t)]/12∆tf ′′ (t) =[−f(t +2∆t)+16f(t +∆t) − 30f(t)+16f(t − ∆t) − f(t − 2∆t)]/12∆t 2f ′′′ (t) =[−f(t +3∆t)+8f(t +2∆t) − 13f(t +∆t)+13f(t − ∆t) − 8f(t − 2∆t)+f(t − 3∆t)]/8∆t 3f ′′′′ (t) =[−f(t +3∆t)+12f(t +2∆t) − 39f(t +∆t)+56f(t)−39f(t − ∆t)+12f(t − 2∆t) − f(t − 3∆t)]/6∆t 4Table 3: Fourth-order accurate center-difference formulas.By using the mean-value theorem of calculus, we find f ′′′ (c) = (f ′′′ (c 1 )+f ′′′ (c 2 ))/2. Upon dividing the above expression by 2∆t and rearranging, wefind the following expression for the first derivative:df (t) f(t +∆t) − f(t − ∆t)= − ∆t2 d 3 f(c)dt2∆t6 dt 3 (1.5.7)where the last term is the truncation error associated with the approximation ofthe first derivative using this particular Taylor series generated expression. Notethat the truncation error in this case is O(∆t 2 ). We could improve on this bycontinuing our Taylor expansion and truncating it at higher orders in ∆t. Thiswould lead to higher accuracy schemes. Further, we could also approximatethe second, third, fourth, and higher derivatives using this technique.Itisalsopossible to generate backward and forward difference schemes byusingpointsonly behind or in front of the current point respectively. Tables 2-4 summarizethe second-order and fourth-order central difference schemes along with theforward- and backward-difference formulas which are accurate to second-order.To solve the simplified linear boundary value problem above which is accurateto second order, we use table 2 for the second and first derivatives. Theboundary value problem then becomesy(t+∆t)−2y(t)+y(t−∆t)∆t 2= p(t) y(t+∆t)−y(t−∆t) +q(t)y(t)+r(t) (1.5.8)2∆twith the boundary conditions y(a) =α and y(b) =β. We can rearrange thisexpression to read[1 − ∆t2 p(t) ]y(t +∆t) − [ 2+∆t 2 q(t) ] y(t)+[1+ ∆t2]y(t − ∆t) =∆t 2 r(t) .(1.5.9)

AMATH 581 ( c○J. N. Kutz) 31O(∆t 2 )forward-andbackward-differenceschemesf ′ (t) =[−3f(t)+4f(t +∆t) − f(t +2∆t)]/2∆tf ′ (t) =[3f(t) − 4f(t − ∆t)+f(t − 2∆t)]/2∆tf ′′ (t) =[2f(t) − 5f(t +∆t)+4f(t +2∆t) − f(t +3∆t)]/∆t 3f ′′ (t) =[2f(t) − 5f(t − ∆t)+4f(t − 2∆t) − f(t − 3∆t)]/∆t 3Table 4: Second-order accurate forward- and backward-difference formulas.We discretize the computational domain and denote t 0 = a to be the left boundarypoint and t N = b to be the right boundary point. This gives the boundaryconditionsy(t 0 )=y(a) =αy(t N )=y(b) =β.(1.5.10a)(1.5.10b)The remaining N − 1pointscanberecastasamatrixproblemAx = b whereand⎡A=⎢⎣⎡x=⎢⎣2+∆t 2 q(t 1 )−1+ ∆t2 p(t 1 ) 0 ··· 0−1− ∆t2 p(t 2 ) 2+∆t2 q(t 2 ) −1+ ∆t2 p(t 2 ) 0 ··· . ..0...y(t 1 )y(t 2 )..y(t N−2 )y(t N−1 ). . .. . .. . ...0. . .. . . −1+ ∆t2 p(t N−2 )0 ··· 0 −1− ∆t2 p(t N−1 ) 2+∆t2 q(t N−1 )⎤⎥⎦⎡b=⎢⎣−∆t 2 r(t 1 )+(1+∆tp(t 1 )/2)y(t 0 )−∆t 2 r(t 2 )..−∆t 2 r(t N−2 )−∆t 2 r(t N−1 )+(1− ∆tp(t N−1 )/2)y(t N )⎤⎥⎦(1.5.11)(1.5.12)Thus the solution can be found by a direct solve of the linear system of equations.⎤.⎥⎦

AMATH 581 ( c○J. N. Kutz) 32Nonlinear SystemsAsimilarsolutionprocedurecanbecarriedoutfornonlinearsystems. However,difficulties arise from solving the resulting set of nonlinear algebraicequations.We can once again consider the general differential equation and expand withsecond-order accurate schemes:y ′′ =f(t, y, y ′ ) → y(t+∆t)−2y(t)+y(t−∆t)∆t 2 =f(t, y(t), y(t+∆t)−y(t−∆t)2∆t(1.5.13)We discretize the computational domain and denote t 0 = a to be the left boundarypoint and t N = b to be the right boundary point. Considering again thesimplified boundary conditions y(t 0 )=y(a) =α and y(t N )=y(b) =β givesthe following nonlinear system for the remaining N − 1points.2y 1 − y 2 − α +∆t 2 f(t 1 ,y 1 , (y 2 − α)/2∆t) =0−y 1 +2y 2 − y 3 +∆t 2 f(t 2 ,y 2 , (y 3 − y 1 )/2∆t) =0.−y N−3 +2y N−2 −y N−1 +∆t 2 f(t N−2 ,y N−2 , (y N−1 −y N−3 )/2∆t) =0−y N−2 +2y N−1 − β +∆t 2 f(t N−1 ,y N−1 , (β − y N−2 )/2∆t) =0.This (N − 1) × (N − 1) nonlinear system of equations can be very difficult tosolve and imposes a severe constraint on the usefulness of the scheme. However,there may be no other way of solving the problem and a solution to these systemof equations must be computed. Further complicating the issue is the fact thatfor nonlinear systems such as these, there are no guarantees about the existenceor uniqueness of solutions. The best approach is to use a relaxation schemewhich is based upon Newton or Secant method iterations.Solving Nonlinear Systems: Newton-Raphson IterationThe only built-in MATLAB command which solves nonlinear system of equationsis FSOLVE. However, this command is now packaged within theoptimizationtoolbox. Most users of MATLAB do not have access to this toolboxand alternatives must be sought. We therefore develop the basic ideas of theNewton-Raphson Iteration method, commonly known as a Newton’s method.We begin by considering a single nonlinear equation).f(x r )=0 (1.5.15)where x r is the root to the equation and the value being sought. We wouldliketodevelop a scheme which systematically determines the value of x r .TheNewton-Raphson method is an iterative scheme which relies on an initial guess, x 0 ,forthe value of the root. From this guess, subsequent guesses are determineduntil

AMATH 581 ( c○J. N. Kutz) 33slope=rise/run=(0-f( ) )/( - )x n x n+1 x nx 1xnx 3 x 0 x 4 x 2x n+1f( x n+1 )f( x n )(a)(b)Figure 11: Construction and implementation of the Newton-Raphson iterationformula. In (a), the slope is the determining factor in deriving the Newton-Raphson formula. In (b), a graphical representation of the iteration scheme isgiven.the scheme either converges to the root x r or the scheme diverges and anotherinitial guess is used. The sequence of guesses (x 0 ,x 1 ,x 2 ,...)isgeneratedfromthe slope of the function f(x). The graphical procedure is illustrated in Fig. 11.In essence, everything relies on the slope formula as illustrated in Fig. 11(a):slope = df (x n)dx= riserun = 0 − f(x n)x n+1 − x n. (1.5.16)Rearranging this gives the Newton-Raphson iterative relationx n+1 = x n − f(x n)f ′ (x n ) . (1.5.17)Agraphicalexampleofhowtheiterationprocedureworksisgiven in Fig. 11(b)where a sequence of iterations is demonstrated. Note that the schemefailsiff ′ (x n )=0sincethentheslopelineneverintersectsy =0. Further,forcertainguesses the iterations may diverge. Provided the initial guess is sufficiently close,the scheme usually will converge. Conditions for convergence can be found inBurden and Faires [5].The Newton method can be generalized for system of nonlinear equations.The details will not be discussed here, but the Newton iteration scheme is similarto that developed for the single function case. Given a system:⎡F(x n )= ⎢⎣f 1 (x 1 ,x 2 ,x 3 ,...,x N )f 2 (x 1 ,x 2 ,x 3 ,...,x N ).f N (x 1 ,x 2 ,x 3 ,...,x N )⎤⎥⎦ = 0 , (1.5.18)

AMATH 581 ( c○J. N. Kutz) 34the iteration scheme iswhereand J(x n )istheJacobianmatrixx n+1 = x n + ∆x n (1.5.19)J(x n )∆x n = −F(x n ) (1.5.20)⎡⎤f 1x1 f 1x2 ··· f 1xNf 2x1 f 2x2 ··· f 2xNJ(x n )= ⎢ . . . ⎥⎣ . . . ⎦f Nx1 f Nx2 ··· f NxN(1.5.21)This algorithm relies on initially guessing values for x 1 ,x 2 ,...,x N . As before,there is no guarantee that the algorithm will converge. Thus agoodinitialguessis critical to its success. Further, the determinant of the Jacobian cannot equalzero, det J(x n ) ≠ 0, inorderforthealgorithmtowork.2 Finite Difference MethodsFinite difference methods are based exclusively on Taylor expansions. They areone of the most powerful methods available since they are relatively easy toimplement, can handle fairly complicated boundary conditions, and allow forexplicit calculations of the computational error. The result of discretizing anygiven problem is the need to solve a large linear system of equations or perhapsmanipulate large, sparse matrices. All this will be dealt with in the followingsections.2.1 Finite difference discretizationTo discuss the solution of a given problem with the finite difference method,we consider a specific example from atmospheric sciences. The quasi-twodimensionalmotion of the atmosphere can be modeled by the advection-diffusionbehavior for the vorticity ω(x, y, t) which is coupled to the streamfunctionψ(x, y, t):where∂ω∂t +[ψ, ω] =ν∇2 ω∇ 2 ψ = ω[ψ, ω] = ∂ψ ∂ω∂x ∂y − ∂ψ ∂ω∂y ∂x(2.1.1a)(2.1.1b)(2.1.2)

AMATH 581 ( c○J. N. Kutz) 35and ∇ 2 = ∂x 2 +∂2 y is the two dimensional Laplacian. Note that this equation hasboth an advection component (hyperbolic) from [ψ, ω] andadiffusioncomponent(parabolic) from ν∇ 2 ω.Wewillassumethatwearegiventheinitialvalueof the vorticityω(x, y, t =0)=ω 0 (x, y) . (2.1.3)Additionally, we will proceed to solve this problem with periodic boundaryconditions. This gives the following set of boundary conditionsω(−L, y, t) =ω(L, y, t)ω(x, −L, t) =ω(x, L, t)ψ(−L, y, t) =ψ(L, y, t)ψ(x, −L, t) =ψ(x, L, t)(2.1.4a)(2.1.4b)(2.1.4c)(2.1.4d)where we are solving on the computational domain x ∈ [−L, L] andy ∈ [−L, L].Basic Algorithm StructureBefore discretizing the governing partial differential equation, it is important toclarify what the basic solution procedure will be. Two physical quantities needto be solved as functions of time:ψ(x, y, t) streamfunction (2.1.5a)ω(x, y, t) vorticity. (2.1.5b)We are given the initial vorticity ω 0 (x, y) andperiodicboundaryconditions.The solution procedure is as follows:1. Elliptic Solve: Solve the elliptic problem ∇ 2 ψ = ω 0 to find the streamfunctionat time zero ψ(x, y, t =0)=ψ 0 .2. Time-Stepping: Given initial ω 0 and ψ 0 ,solvetheadvection-diffusionproblem by time-stepping with a given method. The Euler method isillustrated belowω(x, y, t +∆t) =ω(x, y, t)+∆t ( ν∇ 2 ω(x, y, t) − [ψ(x, y, t),ω(x, y, t)] )This advances the solution ∆t into the future.3. Loop: With the updated value of ω(x, y, ∆t), we can repeat the processby again solving for ψ(x, y, ∆t) andupdatingthevorticityonceagain.This gives the basic algorithmic structure which must be implemented in orderto generate the solution for the vorticity and streamfunction as functions oftime. It only remains to discretize the problem and solve.

AMATH 581 ( c○J. N. Kutz) 36Step 1: Elliptic SolveWe begin by discretizing the elliptic solve problem for the streamfunction ψ(x, y, t).The governing equation in this case is∇ 2 ψ = ∂2 ψ∂x 2 + ∂2 ψ∂y 2 = ω (2.1.6)Using the central difference formulas of Sec. 1.5 reduces the governing equationto a set of linearly coupled equations. In particular, we find for a second-orderaccurate central difference scheme that the elliptic equation reduces to:ψ(x +∆x, y, t) − 2ψ(x, y, t)+ψ(x − ∆x, y, t)∆x 2 (2.1.7)ψ(x, y +∆y, t) − 2ψ(x, y, t)+ψ(x, y − ∆y, t)+∆y 2 = ω(x, y, t)Thus the solution at each point depends upon itself and four neighboring points.This creates a five point stencil for solving this equation. Figure 12 illustratesthe stencil which arises from discretization. For convenience we denoteψ mn = ψ(x m ,y n ) . (2.1.8)By letting ∆x 2 =∆y 2 = δ 2 ,thediscretizedequationsreduceto−4ψ mn + ψ (m−1)n + ψ (m+1)n + ψ m(n−1) + ψ m(n+1) = δ 2 ω mn (2.1.9)with periodic boundary conditions imposing the following constraintsψ 1n = ψ (N+1)nψ m1 = ψ m(N+1)(2.1.10a)(2.1.10b)where N +1is thetotalnumberofdiscretizationpointsinthe computationaldomain in both the x and y directions.As a simple example, consider the four point system for which N =4. Forthis case, we have the following sets of equations−4ψ 11 + ψ 41 + ψ 21 + ψ 14 + ψ 12 = δ 2 ω 11−4ψ 12 + ψ 42 + ψ 22 + ψ 11 + ψ 13 = δ 2 ω 12.−4ψ 21 + ψ 11 + ψ 31 + ψ 24 + ψ 22 = δ 2 ω 21..which results in the sparse matrix (banded matrix) systemAψ = δ 2 ω (2.1.12)

AMATH 581 ( c○J. N. Kutz) 37n=N+1Δ xn=Nn=N-1Δ yyψ mnψm(n-1)ψ (m-1)nψ m(n+1)ψ (m+1)nn=3n=2n=1m=1 m=2 m=3m=M-1 m=M m=M+1xFigure 12: Discretization stencil for solving for the streamfunction with secondorderaccurate central difference schemes. Note that ψ mn = ψ(x m ,y n ).where⎡A=⎢⎣and−4 1 0 1 1 0 0 0 0 0 0 0 1 0 0 01 −4 1 0 0 1 0 0 0 0 0 0 0 1 0 00 1 −4 1 0 0 1 0 0 0 0 0 0 0 1 01 0 1 −4 0 0 0 1 0 0 0 0 0 0 0 11 0 0 0 −4 1 0 1 1 0 0 0 0 0 0 00 1 0 0 1 −4 1 0 0 1 0 0 0 0 0 00 0 1 0 0 1 −4 1 0 0 1 0 0 0 0 00 0 0 1 1 0 1 −4 0 0 0 1 0 0 0 00 0 0 0 1 0 0 0 −4 1 0 1 1 0 0 00 0 0 0 0 1 0 0 1 −4 1 0 0 1 0 00 0 0 0 0 0 1 0 0 1 −4 1 0 0 1 00 0 0 0 0 0 0 1 1 0 1 −4 0 0 0 11 0 0 0 0 0 0 0 1 0 0 0 −4 1 0 10 1 0 0 0 0 0 0 0 1 0 0 1 −4 1 00 0 1 0 0 0 0 0 0 0 1 0 0 1 −4 10 0 0 1 0 0 0 0 0 0 0 1 1 0 1 −4⎤⎥⎦(2.1.13)ψ =(ψ 11 ψ 12 ψ 13 ψ 14 ψ 21 ψ 22 ψ 23 ψ 24 ψ 31 ψ 32 ψ 33 ψ 34 ψ 41 ψ 42 ψ 43 ψ 44 ) T(2.1.14a)

AMATH 581 ( c○J. N. Kutz) 38ω =δ 2 (ω 11 ω 12 ω 13 ω 14 ω 21 ω 22 ω 23 ω 24 ω 31 ω 32 ω 33 ω 34 ω 41 ω 42 ω 43 ω 44 ) T .(2.1.14b)Any matrix solver can then be used to generate the values of thetwo-dimensionalstreamfunction which are contained completely in the vector ψ.Step 2: Time-SteppingAfter generating the matrix A and the value of the streamfunction ψ(x, y, t),we use this updated value along with the current value of the vorticity to take atime step ∆t into the future. The appropriate equation is the advection-diffusionevolution equation:∂ω∂t +[ψ, ω] =ν∇2 ω. (2.1.15)Using the definition of the bracketed term and the Laplacian, this equation is∂ω∂t = ∂ψ ∂ω∂y ∂x − ∂ψ (∂ω ∂ 2 )∂x ∂y + ν ω∂x 2 + ∂2 ω∂y 2 . (2.1.16)Second order central-differencing discretization then yields( )( )∂ω ψ(x, y+∆y, t)−ψ(x, y−∆y, t) ω(x+∆x, y, t)−ω(x−∆x, y, t)=∂t2∆y2∆x( )( )ψ(x+∆x, y, t)−ψ(x−∆x, y, t) ω(x, y+∆y, t)−ω(x, y−∆y, t)−2∆x2∆y{ ω(x+∆x, y, t)−2ω(x, y, t)+ω(x−∆x, y, t)+ν∆x 2}ω(x, y+∆y, t)−2ω(x, y, t)+ω(x, y−∆y, t)+∆y 2 . (2.1.17)This is simply a large system of differential equations which can be steppedforward in time with any convenient time-stepping algorithm suchas4thorderRunge-Kutta. In particular, given that there are N +1 points and periodicboundary conditions, this reduces the system of differential equationstoanN ×N coupled system. Once we have updated the value of the vorticity, we mustagain update the value of streamfunction to once again update thevorticity.This loop continues until the solution at the desired future time is achieved.Figure 13 illustrates how the five-point, two-dimensional stencil advances thesolution.The behavior of the vorticity is illustrated in Fig. 14 where the solution isadvanced for eight time units. The initial condition used in this simulation is)ω 0 = ω(x, y, t =0)=exp(−2x 2 − y2. (2.1.18)20This stretched Gaussian is seen to rotate while advecting and diffusingvorticity.Multiple vortex solutions can also be considered along with oppositely signedvortices.

AMATH 581 ( c○J. N. Kutz) 39ωmnt=2 Δtω mnt= Δtt=0ω mnFigure 13: Discretization stencil resulting from center-differencing of theadvection-diffusion equations. Note that for explicit stepping schemes the futuresolution only depends upon the present. Thus we are not required to solvealargelinearsystemofequations.2.2 Direct solution methods for Ax=bAcentralconcerninalmostanycomputationalstrategyisafast and efficientcomputational method for achieving a solution of a large system of equationsAx = b. Intryingtorenderacomputationtractable,itiscrucialtominimizethe operations it takes in solving such a system. There are a variety of directmethods for solving Ax = b: Gaussian elimination, LU decomposition, andinverting the matrix A. Inadditiontothesedirectmethods,iterativeschemescan also provide efficient solution techniques. Some basic iterative schemes willbe discussed in what follows.The standard beginning to discussions of solution techniques for Ax = binvolves Gaussian elimination. We will consider a very simple example of a3×3systeminordertounderstandtheoperationcountandnumerical procedure

AMATH 581 ( c○J. N. Kutz) 40t=0 t=1t=2t=3 t=4 t=5t=6 t=7 t=8Figure 14: Time evolution of the vorticity ω(x, y, t) overeighttimeunitswithν =0.001 and a spatial domain x ∈ [−10, 10] and y ∈ [−10, 10]. The initialcondition was a stretched Gaussian of the form ω(x, y, 0) = exp(−2x 2 − y 2 /20).involved in this technique. Thus consider Ax = b with⎛A = ⎝ 1 1 1 ⎞⎛1 2 4 ⎠ b = ⎝ 1−11 3 91⎞⎠ . (2.2.1)The Gaussian elimination procedure begins with the construction of the augmentedmatrix⎡[A|b] = ⎣ 1 1 1 1 ⎤1 2 4 −1 ⎦1 3 9 1

AMATH 581 ( c○J. N. Kutz) 44MATLAB: A \ bGiven the alternatives for solving the linear system Ax = b, itisimportanttoknow how the MATLAB command structure for A \ b works. The following isan outline of the algorithm performed.1. It first checks to see if A is triangular, or some permutation thereof. If itis, then all that is needed is a simple O(N 2 )substitutionroutine.2. It then checks if A is symmetric, i.e. Hermitian or Self-Adjoint. If so, aCholesky factorization is attempted. If A is positive definite, the Choleskyalgorithm is always succesful and takes half the run time of LU factorization.3. It then checks if A is Hessenberg. If so, it can be written as an uppertriangular matrix and solved by a substitution routine.4. If all the above methods fail, then LU factorization is used andtheforwardand backward substitution routines generate a solution.5. If A is not square, a QR (Householder) routine is used to solve the system.6. If A is not square and sparse, a least squares solution using QR factorizationis performed.Note that solving by b = A −1 x is the slowest of all methods, taking 2.5 timeslonger or more than A \ b. It is not recommended. However, just like LUfactorization, once the inverse is known it need not be calculated again. Caremust also be taken when the det A ≈ 0, i.e. the matrix is ill-conditioned.MATLAB commandsThe commands for executing the linear system solve are as follows• A \ b: Solve the system in the order above.• [L, U] =lu(A): Generate the L and U matrices.• [L, U, P ]=lu(A): Generate the L and U factorization matrices along withthe permutation matrix P .2.3 Iterative solution methods for Ax=bIn addition to the standard techniques of Gaussian elimination or LU decompositionfor solving Ax = b, awiderangeofiterativetechniquesareavailable.These iterative techniques can often go under the name of Krylov space methods[6]. The idea is to start with an initial guess for the solution and developan iterative procedure that will converge to the solution. The simplest example

AMATH 581 ( c○J. N. Kutz) 45of this method is known as a Jacobi iteration scheme. The implementation ofthis scheme is best illustrated with an example. We consider the linear system4x − y + z =74x − 8y + z = −21−2x + y +5z =15.(2.3.1a)(2.3.1b)(2.3.1c)We can rewrite each equation as followsx = 7+y − z4(2.3.2a)21 + 4x + zy =8(2.3.2b)15 + 2x − yz = .5(2.3.2c)To solve the system iteratively, we can define the following Jacobi iterationscheme based on the abovex k+1 = 7+y k − z k4y k+1 = 21 + 4x k + z k8z k+1 = 15 + 2x k − y k5An algorithm is then easily implemented computationally.would follow the structure:1. Guess initial values: (x 0 ,y 0 ,z 0 ).2. Iterate the Jacobi scheme: x k+1 = Ax k .3. Check for convergence: ‖ x k+1 − x k ‖

AMATH 581 ( c○J. N. Kutz) 46k x k y k z k0 1.0 2.0 2.01 1.75 3.375 3.02 1.84375 3.875 3.025. ...15 1.99999993 3.99999985 2.9999993....19 2.0 4.0 3.0Table 5: Convergence of Jacobi iteration scheme to the solution value of(x, y, z) =(2, 4, 3) from the initial guess (x 0 ,y 0 ,z 0 )=(1, 2, 2).k x k y k z k0 1.0 2.0 2.01 -1.5 3.375 5.02 6.6875 2.5 16.3753 34.6875 8.015625 -17.25.. . .±∞ ±∞ ±∞Table 6: Divergence of Jacobi iteration scheme from the initial guess(x 0 ,y 0 ,z 0 )=(1, 2, 2).To solve the system iteratively, we can define the following Jacobi iterationscheme based on this rearranged set of equationsx k+1 = y k +5z k − 152y k+1 = 21 + 4x k + z k8z k+1 = y k − 4x k +7.(2.3.5a)(2.3.5b)(2.3.5c)Of course, the solution should be exactly as before. However, Table6showsthat applying the iteration scheme leads to a set of values which grow to infinity.Thus the iteration scheme quickly fails.Strictly Diagonal DominantThe difference in the two Jacobi schemes above involves the iteration procedurebeing strictly diagonal dominant. We begin with the definition of strict diagonal

AMATH 581 ( c○J. N. Kutz) 47dominance. A matrix A is strictly diagonal dominant if for each row, the sumof the absolute values of the off-diagonal terms is less than the absolute valueof the diagonal term:N∑|a kk | > |a kj | . (2.3.6)j=1,j≠kStrict diagonal dominance has the following consequence; given a stricly diagonaldominant matrix A, thenAx = b has a unique solution x = p. Jacobiiteration produces a sequence p k that will converge to p for any p 0 . For thetwo examples considered here, this property is crucial. For the first example(2.3.1), we have⎛4 −1 1 ⎞A = ⎝ 4 −8 1⎠ →−2 1 5row 1: |4| > |−1| + |1| =2row 2: |−8| > |4| + |1| =5row 3: |5| > |2| + |1| =3, (2.3.7)which shows the system to be strictly diagonal dominant and guaranteed toconverge. In contrast, the second system (2.3.4) is not stricly diagonal dominantas can be seen from⎛A = ⎝ −2 1 5⎞4 −8 1⎠ →4 −1 1row 1: |−2| < |1| + |5| =6row 2: |−8| > |4| + |1| =5row 3: |1| < |4| + |−1| =5. (2.3.8)Thus this scheme is not guaranteed to converge. Indeed, it diverges to infinity.Modification and Enhancements: Gauss-SeidelIt is sometimes possible to enhance the convergence of a scheme by applyingmodifications to the basic Jacobi scheme. For instance, the Jacobi scheme givenby (2.3.3) can be enhanced by the following modificationsx k+1 = 7+y k − z k4y k+1 = 21 + 4x k+1 + z k8z k+1 = 15 + 2x k+1 − y k+15(2.3.9a)(2.3.9b). (2.3.9c)Here use is made of the supposedly improved value x k+1 in the second equationand x k+1 and y k+1 in the third equation. This is known as the Gauss-Seidelscheme. Table 7 shows that the Gauss-Seidel procedure converges to the solutionin half the number of iterations used by the Jacobi scheme.Unlike the Jacobi scheme, the Gauss-Seidel method is not guaranteed to convergeeven in the case of strict diagonal dominance. Further, theGauss-Seidelmodification is only one of a large number of possible changes to the iteration

AMATH 581 ( c○J. N. Kutz) 48k x k y k z k0 1.0 2.0 2.01 1.75 3.75 2.952 1.95 3.96875 2.98625. . . .10 2.0 4.0 3.0Table 7: Convergence of Gauss-Seidel iteration scheme to thesolutionvalueof(x, y, z) =(2, 4, 3) from the initial guess (x 0 ,y 0 ,z 0 )=(1, 2, 2).scheme which can be implemented in an effort to enhance convergence. It isalso possible to use several previous iterations to achieve convergence. Krylovspace methods [6] are often high end iterative techniques especially developedfor rapid convergence. Included in these iteration schemes are conjugant gradientmethods and generalized minimum residual methods which we will discussand implement [6].Application to Advection-DiffusionWhen discretizing many systems of interest, such as the advection-diffusionproblem, we are left with a system of equations that is naturally geared towarditerative methods. Discretization of the stream function previously yielded thesystem−4ψ mn + ψ (m+1)n + ψ (m−1)n + ψ m(n+1) + ψ m(n−1) = δ 2 ω mn . (2.3.10)The matrix A in this case is represented by the left hand side of the equation.Letting ψ mn be the diagonal term, the iteration procedure yieldsψ k+1mn= ψk (m+1)n + ψk (m−1)n + ψk m(n+1) + ψk m(n−1) − δ2 ω mn4. (2.3.11)Note that the diagonal term had a coefficient of |−4| =4andthesumoftheoffdiagonal elements is |1| + |1| + |1| + |1| =4. Thusthesystemisattheborderlineof being diagonally dominant. So although convergence is not guaranteed,itishighly likely that we could get the Jacobi scheme to converge.Finally, we consider the operation count associated with the iterationmethods.This will allow us to compare this solution technique with Gaussian eliminationand LU decomposition. The following basic algorithmic steps are involved:1. Update each ψ mn which costs N operations times the number of non-zerodiagonals D.

AMATH 581 ( c○J. N. Kutz) 492. For each ψ mn ,performtheappropriateadditionandsubtractions.Inthiscase there are 5 operations.3. Iterate until the desired convergence which costs K operations.Thus the total number of operations is O(N ·D·5·K). If the number of iterationsK can be kept small, then iteration provides a viable alternative to the directsolution techniques.2.4 Fast-Poisson Solvers: the Fourier TransformOther techniques exist for solving many computational problems which are notbased upon the standard Taylor series discretization. For instance, we haveconsidered solving the streamfunction equation∇ 2 ψ = ω (2.4.1)by discretizing in both the x and y directions and solving the associated linearproblem Ax = b. At best, we can use a factorization scheme to solve thisproblem in O(N 2 )operations.Althoughiterationschemeshavethepossibilityof outperforming this, it is not guaranteed.Another alternative is to use the Fast-Fourier Transform (FFT). The FFT isan integral transform defined over the entire line x ∈ [−∞, ∞]. Given computationalpracticalities, however, we transform over a finite domain x ∈ [−L, L] andassume periodic boundary conditions due to the oscillatory behavior of the kernelof the Fourier transform. The Fourier transform and its inverse are definedasF (k) = √ 1 ∫ ∞e ikx f(x)dx(2.4.2a)2π−∞f(x) = 1 √2π∫ ∞−∞e −ikx F (k)dk .(2.4.2b)There are other equivalent definitions. However, this definition will serve toillustrate the power and functionality of the Fourier transform method. Weagain note that formally, the transform is over the entire real line x ∈ [−∞, ∞]whereas our computational domain is only over a finite domain x ∈ [−L, L].Further, the Kernel of the transform, exp(±ikx), describes oscillatory behavior.Thus the Fourier transform is essentially an eigenfunction expansion over allcontinuous wavenumbers k. And once we are on a finite domain x ∈ [−L, L],the continuous eigenfunction expansion becomes a discrete sum of eigenfunctionsand associated wavenumbers (eigenvalues).Derivative RelationsThe critical property in the usage of Fourier transforms concerns derivativerelations. To see how these properties are generated, we begin by considering

AMATH 581 ( c○J. N. Kutz) 50the Fourier transform of f ′ (x).f(x). ̂ Thus we findWe denote the Fourier transform of f(x) asf ̂ ′ (x) = √ 1 ∫ ∞2π−∞e ikx f ′ (x)dx = f(x)e ikx | ∞ −∞ −Assuming that f(x) → 0asx →±∞results inf ̂ ′ (x) =− √ ik ∫ ∞2π−∞√ ik ∫ ∞2π−∞e ikx f(x)dx . (2.4.3)e ikx f(x)dx = −ik ̂f(x) . (2.4.4)Thus the basic relation ̂f ′ = −ik ̂f is established. It is easy to generalize thisargument to an arbitrary number of derivatives. The final result is the followingrelation between Fourier transforms of the derivative and the Fourier transformitself̂f (n) =(−ik) n ̂f . (2.4.5)This property is what makes Fourier transforms so useful and practical.As an example of the Fourier transform, consider the following differentialequationy ′′ − ω 2 y = −f(x) x ∈ [−∞, ∞] . (2.4.6)We can solve this by applying the Fourier transform to both sides. This givesthe following reductionŷ ′′ − ω 2 ŷ = − ̂f−k 2 ŷ − ω 2 ŷ = − ̂f(k 2 + ω 2 )ŷ = ̂f̂fŷ =k 2 + ω 2 . (2.4.7)To find the solution y(x), we invert the last expression above to yieldy(x) = 1 √2π∫ ∞e −ikx−∞̂fk 2 dk . (2.4.8)+ ω2 This gives the solution in terms of an integral which can be evaluated analyticallyor numerically.The Fast Fourier TransformThe Fast Fourier transform routine was developed specifically to perform theforward and backward Fourier transforms. In the mid 1960s, Cooley and Tukeydeveloped what is now commonly known as the FFT algorithm [7]. Their algorithmwas named one of the top ten algorithms of the 20th century for one

AMATH 581 ( c○J. N. Kutz) 51reason: the operation count for solving a system dropped to O(N log N). ForN large, this operation count grows almost linearly like N. Thusitrepresentsagreat leap forward from Gaussian elimination and LU decomposition. The keyfeatures of the FFT routine are as follows:1. It has a low operation count: O(N log N).2. It finds the transform on an interval x ∈ [−L, L]. Since the integrationKernel exp(ikx) isoscillatory,itimpliesthatthesolutionsonthisfiniteinterval have periodic boundary conditions.3. The key to lowering the operation count to O(N log N) isindiscretizingthe range x ∈ [−L, L] into2 n points, i.e. the number of points should be2, 4, 8, 16, 32, 64, 128, 256, ···.4. The FFT has excellent accuracy properties, typically well beyondthatofstandard discretization schemes.We will consider the underlying FFT algorithm in detail at a later time. Formore information at the present, see [7] for a broader overview.The StreamfunctionThe FFT algorithm provides a fast and efficient method for solving the streamfunctionequation∇ 2 ψ = ω (2.4.9)given the vorticity ω. Intwodimensions,thisequationisequivalentto∂ 2 ψ∂x 2 + ∂2 ψ= ω. (2.4.10)∂y2 Denoting the Fourier transform in x as ̂ f(x) andtheFouriertransforminy as˜g(y), we transform the equation. We begin by transforming in x:̂∂ 2 ψ∂x 2 + ̂∂ 2 ψ∂y 2 = ̂ω → −k2 ̂ψ x + ∂2 ̂ψ= ̂ω, (2.4.11)∂y2 where k x are the wavenumbers in the x direction. Transforming now in the ydirection gives−k 2 x˜̂ψ +This can be rearranged to obtain the final result˜∂ 2 ̂ψ∂y 2 = ˜̂ω → −kx2 ˜̂ψ + −ky2 ˜̂ψ = ˜̂ω. (2.4.12)˜̂ψ = −˜̂ωkx 2 + ky2 . (2.4.13)

AMATH 581 ( c○J. N. Kutz) 52The remaining step is to inverse transform in x and y to get back to the solutionψ(x, y).There is one mathematical difficulty which must be addressed. The streamfunctionequation with periodic boundary conditions does not have a uniquesolution. Thus if ψ 0 (x, y, t) isasolution,soisψ 0 (x, y, t) +c where c is anarbitrary constant. When solving this problem with FFTs, the FFTwillarbitrarilyadd a constant to the solution. Fundamentally, we are onlyinterestedin derivatives of the streamfunction. Therefore, this constant is inconsequential.When solving with direct methods for Ax = b, thenonuniquenessgivesasingular matrix A. ThussolvingwithGaussianelimination,LUdecompositionor iterative methods is problematic. But since the arbitray constant does notmatter, we can simply pin the streamfunction to some prescribed value on ourcomputational domain. This will fix the constant c and give a unique solutionto Ax = b. For instance, we could impose the following constraint conditionψ(−L, −L, t) =0. Suchaconditionpinsthevalueofψ(x, y, t) atthelefthandcorner of the computational domain and fixes c.MATLAB commandsThe commands for executing the Fast Fourier Transform and its inverseareasfollows• fft(x): Forward Fourier transform a vector x.• ifft(x): Inverse Fourier transform a vector x.2.5 Comparison of solution techniques for Ax=b: rules ofthumbThe practical implementation of the mathematical tools available in MATLABis crucial. This lecture will focus on the use of some of the more sophisticatedroutines in MATLAB which are cornerstones to scientific computing. Includedin this section will be a discussion of the Fast Fourier Transform routines (fft, ifft,fftshift, ifftshift, fft2, ifft2), sparse matrix construction (spdiag, spy), and highend iterative techniques for solving Ax = b (bicgstab, gmres). These routinesshould be studied carefully since they are the building blocks of any seriousscientific computing code.Fast Fourier Transform: FFT, IFFT, FFTSHIFT, IFFTSHIFT2The Fast Fourier Transform will be the first subject discussed. Its implementationis straightforward. Given a function which has been discretized with 2 npoints and represented by a vector x, theFFTisfoundwiththecommandfft(x). Asidefromtransformingthefunction,thealgorithmassociated with theFFT does three major things: it shifts the data so that x ∈ [0,L] → [−L, 0]

AMATH 581 ( c○J. N. Kutz) 53and x ∈ [−L, 0] → [0,L], additionally it multiplies every other mode by −1, andit assumes you are working on a 2π periodic domain. These properties are aconsequence of the FFT algorithm discussed in detail at a later time.To see the practical implications of the FFT, we consider the transform ofaGaussianfunction. Thetransformcanbecalculatedanalytically so that wehave the exact relations:f(x) =exp(−αx 2 ) → ̂f(k) = 1 √2αexpAsimpleMATLABcodetoverifythiswithα =1isasfollows) (− k2. (2.5.1)4αclear all; close all;% clear all variables and figuresL=20;n=128;% define the computational domain [-L/2,L/2]% define the number of Fourier modes 2^nx2=linspace(-L/2,L/2,n+1); % define the domain discretizationx=x2(1:n); % consider only the first n points: periodicityu=exp(-x.*x); % function to take a derivative ofut=fft(u);% FFT the functionutshift=fftshift(ut); % shift FFTfigure(1), plot(x,u) % plot initial gaussianfigure(2), plot(abs(ut)) % plot unshifted transformfigure(3), plot(abs(utshift)) % plot shifted transformThe second figure generated by this script shows how the pulse is shifted. Byusing the command fftshift, wecanshiftthetransformedfunctionbacktoitsmathematically correct positions as shown in the third figure generated. However,before inverting the transformation, it is crucial that the transform isshifted back to the form of the second figure. The command ifftshift does this.In general, unless you need to plot the spectrum, it is better not to deal withthe fftshift and ifftshift commands. A graphical representation of the fft procedureand its shifting properties is illustrated in Fig. 15 where a Gaussian istransformed and shifted by the fft routine.To take a derivative, we need to calculate the k values associated with thetransformation. The following example does this. Recall that the FFT assumesa2π periodic domain which gets shifted. Thus the calculation of the k valuesneeds to shift and rescale to the 2π domain. The following example differentiatesthe function f(x) =sech(x) threetimes. Thefirstderivativeiscomparedwiththe analytic value of f ′ (x) =−sech(x)tanh(x).clear all; close all;% clear all variables and figures

AMATH 581 ( c○J. N. Kutz) 54u=exp(−x 2 )10.80.60.40.2real[fft(u)]151050−5−100−10 −5 0 5 10x15−15−64 −32 0 32 64Fourier modes1510real[fftshift(u)]50−5abs(fftshift(u))105−10−15−64 −32 0 32 64Fourier modes0−64 −32 0 32 64Fourier modesFigure 15: Fast Fourier Transform of Gaussian data illustrating the shiftingproperties of the FFT routine. Note that the fftshift command restores thetransform to its mathematically correct, unshifted state.L=20;n=128;% define the computational domain [-L/2,L/2]% define the number of Fourier modes 2^nx2=linspace(-L/2,L/2,n+1); % define the domain discretizationx=x2(1:n); % consider only the first n points: periodicityu=sech(x); % function to take a derivative ofut=fft(u); % FFT the functionk=(2*pi/L)*[0:(n/2-1) (-n/2):-1]; % k rescaled to 2pi domainut1=i*k.*ut;ut2=-k.*k.*ut;ut3=-i*k.*k.*k.*ut;% first derivative% second derivative% third derivativeu1=ifft(ut1); u2=ifft(ut2); u3=ifft(ut3); % inverse transformu1exact=-sech(x).*tanh(x); % analytic first derivative

AMATH 581 ( c○J. N. Kutz) 55figure(1)plot(x,u,’r’,x,u1,’g’,x,u1exact,’go’,x,u2,’b’,x,u3,’c’) % plotThe routine accounts for the periodic boundaries, the correct k values, anddifferentiation. Note that no shifting was necessary since we constructedthekvalues in the shifted space.For transforming in higher dimensions, a couple of choices inMATLABarepossible. For 2D transformations, it is recommended to use the commandsfft2 and ifft2. These will transform a matrix A, whichrepresentsdatainthex and y direction respectively, along the rows and then columns. For higherdimensions, the fft command can be modified to fft(x,[],N) where N is thenumber of dimensions.Sparse Matrices: SPDIAG, SPYUnder discretization, most physical problems yield sparse matrices, i.e. matriceswhich are largely composed of zeros. For instance, the matrices (1.5.11) and(2.1.13) are sparse matrices generated under the discretization of a boundaryvalue problem and Poisson equation respectively. The spdiag command allowsfor the construction of sparse matrices in a relatively simple fashion. The sparsematrix is then saved using a minimal amount of memory and all matrix operationsare conducted as usual. The spy command allows you to look at thenonzero components of the matrix structure. As an example, we constructthematrix given by (2.1.13) for the case of N =5inboththex and y directions.clear all; close all;% clear all variables and figuresm=5;n=m*m;% N value in x and y directions% total size of matrixe0=zeros(n,1);e1=ones(n,1);% vector of zeros% vector of onese2=e1; % copy the one vectore4=e0; % copy the zero vectorfor j=1:me2(m*j)=0; % overwrite every m^th value with zeroe4(m*j)=1; % overwirte every m^th value with oneende3(2:n,1)=e2(1:n-1,1); e3(1,1)=e2(n,1); % shift to correcte5(2:n,1)=e4(1:n-1,1); e5(1,1)=e4(n,1); % positions% place diagonal elements

AMATH 581 ( c○J. N. Kutz) 56Figure 16: Sparse matrix structure for the Laplacian operator using second orderdiscretization. The output is generated via the spy command in MATLAB.matA=spdiags([e1 e1 e5 e2 -4*e1 e3 e4 e1 e1], ...[-(n-m) -m -m+1 -1 0 1 m-1 m (n-m)],n,n);spy(matA) % view the matrix structureThe appropriate construction of the sparse matrix is critical to solving anyproblem. It is essential to carefully check the matrix for accuracy before usingit in an application. As can be seen from this example, it takes abitofworkto get the matrix correct. However, once constructed properly, it can be usedwith confidence in all other matrix applications and operations.Iterative Methods: BICGSTAB, GMRESIterative techniques need also to be considered. There are a wide variety ofbuilt-in iterative techniques in MATLAB. Two of the more promising methodsare discussed here: the bi-conjugate stabilized gradient method (bicgstab) andthe generalized minimum residual method (gmres). Both are easily implementedin MATLAB.Recall that these iteration methods are for solving the linear system Ax = b.The basic call to the generalized minimum residual method is

AMATH 581 ( c○J. N. Kutz) 57>>x=gmres(A,b);Likewise, the bi-conjugate stabilized gradient method is called by>>x=bicgstab(A,b);It is rare that you would use these commands without options. The iterationscheme stops upon convergence, failure to converge, or when the maximum numberof iterations has been achieved. By default, the initial guess for the iterationprocedure is the zero vector. The default number of maximum iterations is 10,which is rarely enough iterations for the purposes of scientific computing.Thus it is important to understand the different options available with theseiteration techniques. The most general user specified command line for eithergmres or bicgstab is as follows>>[x,flag,relres,iter]=bicgstab(A,b,tol,maxit,M1,M2,x0);>>[x,flag,relres,iter]=gmres(A,b,restart,tol,maxit,M1,M2,x0);We already know that the matrix A and vector b come from the original problem.The remaining parameters are as follows:tol = specified tolerance for convergencemaxit = maximum number of iterationsM1, M2 = pre-conditioning matricesx0 = initial guess vectorrestart = restart of iterations (gmres only)In addition to these input parameters, gmres or bicgstab will give the relativeresidual relres and the number of iterations performed iter. The flag variablegives information on whether the scheme converged in the maximum allowableiterations or not.2.6 Overcoming computational difficultiesIn developing algorithms for any given problem, you should always try to maximizethe use of information available to you about the specific problem. Specifically,a well developed numerical code will make extensive use of analytic informationconcerning the problem. Judicious use of properties of the specificproblem can lead to significant reduction in the amount of computational timeand effort needed to perform a given calculation.Here, various practical issues which may arise in the computational implementationof a given method are considered. Analysis provides a strong foundationfor understanding the issues which can be problematic on acomputationallevel. Thus the focus here will be on details which need to be addressed beforestraightforward computing is performed.

AMATH 581 ( c○J. N. Kutz) 58Streamfunction Equations: NonuniquenessWe return to the consideration of the streamfunction equation∇ 2 ψ = ω (2.6.1)with the periodic boundary conditionsψ(−L, y, t) =ψ(L, y, t)ψ(x, −L, t) =ψ(x, L, t) .(2.6.2a)(2.6.2b)The mathematical problem which arises in solving this Poisson equation hasbeen considered previously. Namely, the solution can only be determinedtoanarbitrary constant. Thus if ψ 0 is a solution to the streamfunction equation, soisψ = ψ 0 + c, (2.6.3)where c is an arbitrary constant. This gives an infinite number of solutions tothe problem. Thus upon discretizing the equation in x and y and formulatingthe matrix formulation of the problem Ax = b, wefindthatthematrixA,which is given by (2.1.13), is singular, i.e. det A =0.Obviously, the fact that the matrix A is singular will create computationalproblems. However, does this nonuniqueness jeopardize the validity of the physicalmodel? Recall that the streamfunction is a fictitious quantity which capturesthe fluid velocity through the quantities ∂ψ/∂x and ∂ψ/∂y. In particular, wehave the x and y velocity componentsu = − ∂ψ∂y(x-component of velocity) (2.6.4a)v = ∂ψ∂x(y-component of velocity) . (2.6.4b)Thus the arbitrary constant c drops out when calculating physically meaningfulquantities. Further, when considering the advection-diffusion equationwhere∂ω∂t +[ψ, ω] =ν∇2 ω (2.6.5)[ψ, ω] = ∂ψ ∂ω∂x ∂y − ∂ψ ∂ω∂y ∂x , (2.6.6)only the derivative of the streamfunction is important, which again removes theconstant c from the problem formulation.We can thus conclude that the nonuniqueness from the constant c does notgenerate any problems for the physical model considered. However, we stillhave the mathematical problem of dealing with a singular matrix. It should benoted that this problem also occurs when all the boundary conditions are of theNeuman type, i.e. ∂ψ/∂n =0wheren denotes the outward normal direction.

AMATH 581 ( c○J. N. Kutz) 59To overcome this problem numerically, we simply observe thatwecanarbitrarilyadd a constant to the solution. Or alternatively, we can pin down thevalue of the streamfunction at a single location in the computational domain.This constraint fixes the arbitrary constant problem and removes the singularityfrom the matrix A. Thus to fix the problem, we can simply pick an arbitrarypoint in our computational domain ψ mn and fix its value. Essentially, this willalter a single component of the sparse matrix (2.1.13). And in fact,thisisthesimplest thing to do. For instance, given the construction of thematrix(2.1.13),we could simply add the following line of MATLAB code:A(1,1)=0;Then the det A ≠0andthematrixcanbeusedinanyofthelinearsolutionmethods. Note that the choice of the matrix component and its value arecompletely arbitrary. However, in this example, if you choose to alter this matrixcomponent, to overcome the matrix singularity you must have A(1, 1) ≠ −4.Fast Fourier Transforms: Divide by ZeroIn addition to solving the streamfunction equation by standard discretizationand Ax = b, wecouldusetheFouriertransformmethod. Inparticular,Fouriertransforming in both x and y reduces the streamfunction equation∇ 2 ψ = ω (2.6.7)to the equation (2.4.13)˜̂ψ = −˜̂ωkx 2 + . (2.6.8)k2 yHere we have denoted the Fourier transform in x as f(x) ̂ andtheFouriertransformin y as ˜g(y). The final step is to inverse transform in x and y to get backto the solution ψ(x, y). It is recommended that the routines fft2 and ifft2 beused to perform the transformations. However, fft and ifft may also be used inloops or with the dimension option set to two.An observation concerning (2.6.8) is that there will be a divide by zero whenk x = k y =0atthezeromode. Twooptionsarecommonlyusedtoovercomethis problem. The first is to modify (2.6.8) so that˜̂ω˜̂ψ = −kx 2 + ky 2 + eps(2.6.9)where eps is the command for generating a machine precision number which ison the order of O(10 −15 ). It essentially adds a round-off to the denominatorwhich removes the divide by zero problem. A second option, which is morehighly recommended, is to redefine the k x and k y vectors associated with thewavenumbers in the x and y directions. Specifically, after defining the k x andk y ,wecouldsimplyaddthecommandline

AMATH 581 ( c○J. N. Kutz) 60kx(1)=10^(-6);ky(1)=10^(-6);The values of kx(1) = ky(1) = 0 by default. This would make the values smallbut finite so that the divide by zero problem is effectively removed with only asmall amount of error added to the problem.Sparse Derivative Matrices: Advection TermsThe sparse matrix (2.1.13) represents the discretization of theLaplacianwhichis accurate to second order. However, when calculating the advection termsgiven by[ψ, ω] = ∂ψ ∂ω∂x ∂y − ∂ψ ∂ω∂y ∂x , (2.6.10)only the first derivative is required in both x and y. Associated with each ofthese derivative terms is a sparse matrix which must be calculated in order toevaluate the advection term.The calculation of the x derivative will be considered here. The y derivativecan be calculated in a similar fashion. Obviously, it is crucial that the sparsematrices representing these operations be correct. Consider then the secondorder discretization of∂ω ω(x +∆x, y) − ω(x − ∆x, y)= . (2.6.11)∂x 2∆xUsing the notation developed previously, we define ω(x m ,y n )=ω mn .Thusthediscretization yields∂ω mn∂x= ω (m+1)n − ω (m−1)n, (2.6.12)2∆xwith periodic boundary conditions. The first few terms of this linearsystemareas follows:∂ω 11∂x∂ω 12∂x= ω 21 − ω n12∆x= ω 22 − ω n22∆x..∂ω 21∂x = ω 31 − ω 112∆xFrom these individual terms, the linear system ∂ω/∂x =(1/2∆x)Bω can beconstructed. The critical component is the sparse matrix B. Notethatwehave. .

AMATH 581 ( c○J. N. Kutz) 61used the usual definition of the vector ω.⎛ ⎞ω 11ω 12.ω 1nω =ω 21. (2.6.14)ω 22.⎜ .⎟⎝ ω n(n−1)⎠ω nnIt can be verified then that the sparse matrix B is given by the matrix⎡⎤0 I 0 ··· 0 −I−I 0 I 0 ··· 00 . . . . .. . ..B=.., (2.6.15)0⎢⎥⎣ . ··· 0 −I 0 I ⎦I 0 ··· 0 −I 0where 0 is an n×n zero matrix and I is an n×n identity matrix. Recall that thematrix B is an n 2 ×n 2 matrix. The sparse matrix associated with this operationcan be constructed with the spdiag command. Following the same analysis, they derivative matrix can also be constructed. If we call this matrix C, thentheadvection operator can simply be written as[ψ, ω] = ∂ψ ∂ω∂x ∂y − ∂ψ ∂ω=(Bψ)(Cω) − (Cψ)(Bω) . (2.6.16)∂y ∂xThis forms part of the right hand side of the system of differential equationswhich is then solved using a standard time-stepping algorithm.3 Time and Space Stepping Schemes: Methodof LinesWith the Fourier transform and discretization in hand, we can turntowardsthesolution of partial differential equations whose solutions need to be advancedforward in time. Thus, in addition to spatial discretization, we will need todiscretize in time in a self-consistent way so as to advance the solution to adesired future time. Issues of stability and accuracy are, of course,attheheartof a discussion on time and space stepping schemes.

AMATH 581 ( c○J. N. Kutz) 623.1 Basic time-stepping schemesBasic time-stepping schemes involve discretization in both spaceandtime. Issuesof numerical stability as the solution is propagated in time and accuracyfrom the space-time discretization are of greatest concern for any implementation.To begin this study, we will consider very simple and well known examplesfrom partial differential equations. The first equation we consider is the heat(diffusion) equation∂u∂t = uκ∂2 ∂x 2 (3.1.1)with the periodic boundary conditions u(−L) =u(L). We discretize the spatialderivative with a second-order scheme (see Table 2) so that∂u∂t =κ [u(x +∆x) − 2u(x)+u(x − ∆x)] . (3.1.2)∆x2 This approximation reduces the partial differential equation to a system of ordinarydifferential equations. We have already considered a variety of timesteppingschemes for differential equations and we can apply them directly tothis resulting system.The ODE SystemTo define the system of ODEs, we discretize and define the valuesofthevectoru in the following way.u(−L) =u 1u(−L +∆x) =u 2.u(L − 2∆x) =u n−1u(L − ∆x) =u nu(L) =u n+1 .Recall from periodicity that u 1 = u n+1 . Thus our system of differential equationssolves for the vector ⎛ ⎞u 1u 2u = ⎜ ⎟⎝ . ⎠ . (3.1.3)u nThe governing equation (3.1.2) is then reformulated as the differential equationssystemdudt = κ Au , (3.1.4)∆x2

AMATH 581 ( c○J. N. Kutz) 63where A is given by the sparse matrix⎡⎤−2 1 0 ··· 0 11 −2 1 0 ··· 0. 0 .. . .. . ..A=.., (3.1.5)0⎢⎥⎣ . ··· 0 1 −2 1 ⎦1 0 ··· 0 1 −2and the values of one on the upper right and lower left of the matrix result fromthe periodic boundary conditions.MATLAB implementationThe system of differential equations can now be easily solved with a standardtime-stepping algorithm such as ode23 or ode45. Thebasicalgorithmwouldbeas follows1. Build the sparse matrix A.e1=ones(n,1); % build a vector of onesA=spdiags([e1 -2*e1 e1],[-1 0 1],n,n); % diagonalsA(1,n)=1; A(n,1)=1; % periodic boundaries2. Generate the desired initial condition vector u = u 0 .3. Call an ODE solver from the MATLAB suite. The matrix A, thediffusionconstant κ and spatial step ∆x need to be passed into this routine.[t,y]=ode45(’rhs’,tspan,u0,[],k,dx,A);The function rhs.m should be of the following formfunction rhs=rhs(tspan,u,dummy,k,dx,A)rhs=(k/dx^2)*A*u;4. Plot the results as a function of time and space.The algorithm is thus fairly routine and requires very littleeffortinprogrammingsince we can make use of the standard time-stepping algorithms already availablein MATLAB.

AMATH 581 ( c○J. N. Kutz) 642D MATLAB implementationIn the case of two dimensions, the calculation becomes slightly more difficultsince the 2D data is represented by a matrix and the ODE solvers requireavector input for the initial data. For this case, the governing equation is(∂u ∂ 2 )∂t = κ u∂x 2 + ∂2 u∂y 2 (3.1.6)Discretizing in x and y gives a right hand side which takes on the form of (2.1.7).Provided ∆x =∆y = δ are the same, the system can be reduced to the linearsystemdudt = κ Au , (3.1.7)δ2 where we have arranged the vector u in a similar fashion to (2.1.14) so that⎛u =⎜⎝u 11u 12..u 1nu 21u 22.u n(n−1)u nn⎞, (3.1.8)⎟⎠where we have defined u jk = u(x j ,y k ). The matrix A is a nine diagonal matrixgiven by (2.1.13). The sparse implementation of this matrix is also givenpreviously.Again, the system of differential equations can now be easily solved with astandard time-stepping algorithm such as ode23 or ode45. Thebasicalgorithmfollows the same course as the 1D case, but extra care is taken in arranging the2D data into a vector.1. Build the sparse matrix A (2.1.13).2. Generate the desired initial condition matrix U = U 0 and reshape it to avector u = u 0 . This example considers the case of a simple Gaussian asthe initial condition. The reshape and meshgrid commands are importantfor computational implementation.Lx=20;Ly=20;nx=100;% spatial domain of x% spatial domain of y% number of discretization points in x

AMATH 581 ( c○J. N. Kutz) 65ny=100; % number of discretization points in yN=nx*ny; % elements in reshaped initial conditionx2=linspace(-Lx/2,Lx/2,nx+1); % account for periodicityx=x2(1:nx); % x-vectory2=linspace(-Ly/2,Ly/2,ny+1); % account for periodicityy=y2(1:ny); % y-vector[X,Y]=meshgrid(x,y);U=exp(-X.^2-Y.^2);u=reshape(U,N,1);% set up for 2D initial conditions% generate a Gaussian matrix% reshape into a vector3. Call an ODE solver from the MATLAB suite. The matrix A, thediffusionconstant κ and spatial step ∆x =∆y = dx need to be passed into thisroutine.[t,y]=ode45(’rhs’,tspan,u0,[],k,dx,A);The function rhs.m should be of the following formfunction rhs=rhs(tspan,u,dummy,k,dx,A)rhs=(k/dx^2)*A*u;4. Plot the results as a function of time and space.The algorithm is again fairly routine and requires very little effort in programmingsince we can make use of the standard time-stepping algorithms.Method of LinesFundamentally, these methods use the data at a single slice oftimetogenerateasolution∆t in the future. This is then used to generate a solution 2∆t intothe future. The process continues until the desired future time is achieved.This process of solving a partial differential equation is known as the method oflines. Eachline is the value of the solution at a given time slice. The lines areused to update the solution to a new timeline and progressively generate futuresolutions. Figure 17 depicts the process involved in the method of lines for the1D case. The 2D case was illustrated previously in Fig. 13.

AMATH 581 ( c○J. N. Kutz) 66ttj+1jtime tsolution progessionalong 1D lineslinesttt210xxxxn-2 n-1 n n+1xn+2space xFigure 17: Graphical representation of the progression of the numerical solutionusing the method of lines. Here a second order discretization scheme isconsidered which couples each spatial point with its nearest neighbor.3.2 Time-stepping schemes: explicit and implicit methodsNow that several technical and computational details have been addressed, wecontinue to develop methods for time-stepping the solution into the future.Some of the more common schemes will be considered along with agraphicalrepresentation of the scheme. Every scheme eventually leads toaniterationprocedure which the computer can use to advance the solution in time.We will begin by considering the most simple partial differential equations.Often, it is difficult to do much analysis with complicated equations. Therefore,considering simple equations is not merely an exercise, but rather they aretypically the only equations we can make analytical progress with.As an example, we consider the one-way wave equation∂u∂t = c∂u ∂x . (3.2.1)The simplest discretization of this equation is to first central difference in the x

AMATH 581 ( c○J. N. Kutz) 67m+1um+1 nttmum m mn-1 un un+1Figure 18: Four-point stencil for second-order spatial discretization and Eulertime-stepping of the one-wave wave equation.direction. This yields∂u n∂t= c2∆x (u n+1 − u n−1 ) , (3.2.2)where u n = u(x n ,t). We can then step forward with an Euler time-steppingmethod. Denoting u (m)n = u(x n ,t m ), and applying the method of lines iterationscheme givesu n(m+1) = u (m)n + c∆t ()u (m)n+12∆x− u(m) n−1 . (3.2.3)This simple scheme has the four-point stencil shown in Fig. 18. To illustratemore clearly the iteration procedure, we rewrite the discretized equation in theformu n(m+1) = u (m)n + λ ()u (m)n+12− u(m) n−1(3.2.4)whereλ = c∆t(3.2.5)∆xis known as the CFL (Courant, Friedrichs, and Lewy) condition [8]. Theiterationprocedure assumes that the solution does not change significantly from onetime-step to the next, i.e. u (m)n ≈ u (m+1)n . The accuracy and stability of thisscheme is controlled almost exclusively by the CFL number λ. Thisparameterrelates the spatial and time discretization schemes in (3.2.5). Note that decreasing∆x without decreasing ∆t leads to an increase in λ which can result ininstabilities. Smaller values of ∆t suggest smaller values of λ and improved numericalstability properties. In practice, you want to take ∆x and ∆t as large aspossible for computational speed and efficiency without generating instabilities.There are practical considerations to keep in mind relating to the CFL number.First, given a spatial discretization step-size ∆x, youshouldchoosethetime discretization so that the CFL number is kept in check. Often a givenscheme will only work for CFL conditions below a certain value, thus the importanceof choosing a small enough time-step. Second, if indeed you choose to

AMATH 581 ( c○J. N. Kutz) 68tm+1u nm+1tmtm-1ummn-1 un+1u nm-1Figure 19: Four-point stencil for second-order spatial discretization and centraldifferencetime-stepping of the one-wave wave equation.work with very small ∆t, thenalthoughstabilitypropertiesareimprovedwithalowerCFLnumber,thecodewillalsoslowdownaccordingly.Thus achievinggood stability results is often counter-productive to fast numerical solutions.Central Differencing in TimeWe can discretize the time-step in a similar fashion to the spatial discretization.Instead of the Euler time-stepping scheme used above, we could centraldifference in time using Table 2. Thus after spatial discretization we have∂u n∂t= c2∆x (u n+1 − u n−1 ) , (3.2.6)as before. And using a central difference scheme in time now yieldsu (m+1)n− u (m−1)n2∆tThis last expression can be rearranged to giveu (m+1)n= u (m−1)n= c ()u (m)n+12∆x− u(m) n−1 . (3.2.7)+ λ()u (m)n+1 − u(m) n−1. (3.2.8)This iterative scheme is called leap-frog (2,2) since it is O(∆t 2 )accurateintimeand O(∆x 2 )accurateinspace.ItusesafourpointstencilasshowninFig. 19.Note that the solution utilizes two time slices to leap-frog to the next time slice.Thus the scheme is not self-starting since only one time slice (initialcondition)is given.

AMATH 581 ( c○J. N. Kutz) 69Improved AccuracyWe can improve the accuracy of any of the above schemes by usinghigherordercentral differencing methods. The fourth-order accurate scheme from Table 3gives∂u −u(x +2∆x)+8u(x +∆x) − 8u(x − ∆x)+u(x − 2∆x)= . (3.2.9)∂x 12∆xCombining this fourth-order spatial scheme with second-order central differencingin time gives the iterative scheme[ 4()u (m+1)n = u (m−1)n + λ u (m)n+13 − u(m) n−1 − 1 (u (m)n+26n−2) ]− u(m) . (3.2.10)This scheme, which is based upon a six point stencil, is called leap frog (2,4). Itis typical that for (2,4) schemes, the maximum CFL number for stable computationsis reduced from the basic (2,2) scheme.Lax-WendroffAnother alternative to discretizing in time and space involves a clever use of theTaylor expansion∂u(x, t)u(x, t +∆t) =u(x, t)+∆t + ∆t2 ∂ 2 u(x, t)∂t 2! ∂t 2 + O(∆t 3 ) . (3.2.11)But we note from the governing one-way wave equation∂u∂t = c∂u ∂x ≈ c2∆x (u n+1 − u n−1 ) .Taking the derivative of the equation results in the relation∂ 2 u∂t 2= c∂2 u∂x 2 ≈c∆x 2 (u n+1 − 2u n + u n−1 ) .(3.2.12a)(3.2.13a)These two expressions for ∂u/∂t and ∂ 2 u/∂t 2 can be substituted into the Taylorseries expression to yield the iterative schemeu (m+1)n= u (m)n + λ () (u (m)n+12− u(m) n−1 + λ2u (m)n+12− 2u(m) n)+ u (m)n−1 . (3.2.14)This iterative scheme is similar to the Euler method. However, it introduces animportant stabilizing diffusion term which is proportional to λ 2 .Thisisknownas the Lax-Wendroff scheme. Although useful for this example, it is difficult toimplement in practice for variable coefficient problems. It illustrates, however,the variety and creativity in developing iteration schemes for advancing thesolution in time and space.

AMATH 581 ( c○J. N. Kutz) 70SchemeStabilityForward Euler unstable for all λBackward Euler stable for all λLeap Frog (2,2) stable for λ ≤ 1Leap Frog (2,4) stable for λ ≤ 0.707Table 8: Stability of time-stepping schemes as a function of the CFL number.Backward Euler: Implicit SchemeThe backward Euler method uses the future time for discretizing the spatialdomain. Thus upon discretizing in space and time we arrive at the iterationschemeu (m+1)nThis gives the tridiagonal systemu (m)nwhich can be written in matrix form aswhere= u (m)n + λ ()u (m+1)n+1 − u (m+1)n−1 . (3.2.15)2= − λ 2 u(m+1) n+1 + u n(m+1) + λ 2 u(m+1) n−1 , (3.2.16)Au (m+1) = u (m) (3.2.17)⎛⎞2 −λ ··· 0A = 1 λ 2 −λ ···⎜2 ⎝.. ..⎟. ⎠ . (3.2.18)0 ··· λ 2Thus before stepping forward in time, we must solve a matrix problem. Thiscan severely affect the computational time of a given scheme. The only thingwhich may make this method viable is if the CFL condition is such that muchlarger time-steps are allowed, thus overcoming the limitations imposed by thematrix solve.MacCormack SchemeIn the MacCormack Scheme, the variable coefficient problem of the Lax-Wendroffscheme and the matrix solve associated with the backward Euler are circumventedby using a predictor-corrector method. The computation thus occurs intwo pieces:u (P )n= u (m)n()+ λ u (m)n+1 − u(m) n−1(3.2.19a)

AMATH 581 ( c○J. N. Kutz) 71u (m+1)n = 1 [()]u (m)n + u (P n) + λ u (P )n+12− u(P )n−1 . (3.2.19b)This method essentially combines forward and backward Euler schemessothatwe avoid the matrix solve and the variable coefficient problem.The CFL condition will be discussed in detail in the next section. For now,the basic stability properties of many of the schemes considered here are givenin Table 8. The stability of the various schemes hold only for the one-waywave equation considered here as a prototypical example. Each partial differentialequation needs to be considered and classified individually with regardsto stability.3.3 Stability analysisIn the preceeding lecture, we considered a variety of schemes whichcansolvethe time-space problem posed by partial differential equations. However, it remainsundetermined which scheme is best for implementation purposes. Twocriteria provide a basis for judgement: accuracy and stability. For each scheme,we already know the accuracy due to the discretization error. However, a determinationof stability is still required.To start to understand stability, we once again consider the one-way waveequation∂u∂t = c∂u ∂x . (3.3.1)Using Euler time-stepping and central differencing in space gives the iterationprocedureu (m+1)nwhere λ is the CFL number.von Neumann Analysis= u (m)n + λ ()u (m)n+12− u(m) n−1(3.3.2)To determine the stability of a given scheme, we perform a von Neumann analysis[9]. This assumes the solution is of the formu (m)n = g m exp(iξ n h) ξ ∈ [−π/h, π/h] (3.3.3)where h =∆x is the spatial discretization parameter. Essentially this assumesthe solution can be constructed of Fourier modes. The key then istodeterminewhat happens to g m as m →∞.Twopossibilitiesexistlim m→∞ |g| m →∞ unstable scheme (3.3.4a)lim m→∞ |g| m ≤ 1(< ∞) stable scheme. (3.3.4b)Thus this stability check is very much like that performed for thetime-steppingschemes developed for ordinary differential equations.

AMATH 581 ( c○J. N. Kutz) 72Forward Euler for One-Way Wave EquationThe Euler discretization of the one-way wave equation produces the iterativescheme (3.3.2). Plugging in the ansatz (3.3.3) gives the following relations:g m+1 exp(iξ n h)=g m exp(iξ n h)+ λ 2 (gm exp(iξ n+1 h) − g m exp(iξ n−1 h))g m+1 exp(iξ n h)=g(exp(iξ m n h)+ λ )2 (exp(iξ n+1h) − exp(iξ n−1 h))g =1+ λ 2 [exp(ih(ξ n+1 − ξ n )) − exp(ih(ξ n−1 − ξ n ))] . (3.3.5)Letting ξ n = nζ reduces the equations further toFrom this we can deduce thatg(ζ) =1+ λ [exp(iζh) − exp(−iζh)]2g(ζ) =1+iλ sin(ζh) . (3.3.6)|g(ζ)| = √ g ∗ (ζ)g(ζ) =√1+λ 2 sin 2 ζh. (3.3.7)Thus|g(ζ)| ≥1 → limm→∞ |g|m →∞ unstable scheme . (3.3.8)Thus the forward Euler time-stepping scheme is unstable for any value of λ.The implementation of this scheme will force the solution to infinity due to anumerical instability.Backward Euler for One-Way Wave EquationThe Euler discretization of the one-way wave equation produces the iterativeschemeu (m+1)n = u (m)n + λ ()u (m+1)n+1 − u (m+1)n−1 . (3.3.9)2Plugging in the ansatz (3.3.3) gives the following relations:g m+1 exp(iξ n h)=g m exp(iξ n h)+ λ 2(g m+1 exp(iξ n+1 h) − g m+1 exp(iξ n−1 h) )g =1+ λ 2 g [exp(ih(ξ n+1 − ξ n )) − exp(ih(ξ n−1 − ξ n ))] . (3.3.10)Letting ξ n = nζ reduces the equations further tog(ζ) =1+ λ g[exp(iζh) − exp(−iζh)]2g(ζ) =1+iλg sin(ζh)g[1 − iλ sin(ζh)] = 11g(ζ) =1 − iλ sin ζh . (3.3.11)

AMATH 581 ( c○J. N. Kutz) 73From this we can deduce that|g(ζ)| = √ √g ∗ 1(ζ)g(ζ) =1+λ 2 sin 2 ζh . (3.3.12)Thus|g(ζ)| ≤1 → limm→∞ |g|m ≤ 1 unconditionally stable. (3.3.13)Thus the backward Euler time-stepping scheme is stable for any value of λ. Theimplementation of this scheme will not force the solution to infinity.Lax-Wendroff for One-Way Wave EquationThe Lax-Wendroff scheme is not as transparent as the forward and backwardEuler schemes.u (m+1)n= u (m)n + λ () (u (m)n+12− u(m) n−1 + λ2u (m)n+12− 2u(m) nPlugging in the standard ansatz (3.3.3) gives)+ u (m)n−1 . (3.3.14)g m+1 exp(iξ n h)=g m exp(iξ n h)+ λ 2 gm (exp(iξ n+1 h) − exp(iξ n−1 h))+ λ22 gm (exp(iξ n+1 h) − 2exp(iξ n h)+exp(iξ n−1 h))g =1+ λ 2 [exp(ih(ξ n+1 − ξ n )) − exp(ih(ξ n−1 − ξ n ))]+ λ22 [exp(ih(ξ n+1 − ξ n )) + exp(ih(ξ n−1 − ξ n )) − 2] . (3.3.15)Letting ξ n = nζ reduces the equations further tog(ζ) =1+ λ λ2[exp(iζh) − exp(−iζh)] + [exp(iζh)+exp(−iζh) − 2]2 2g(ζ) =1+iλ sin ζh + λ 2 (cos ζh − 1)g(ζ) =1+iλ sin ζh − 2λ 2 sin 2 (ζh/2) . (3.3.16)This results in the relation|g(ζ)| 2 = λ 4 [ 4sin 4 (ζh/2) ] + λ 2 [ sin 2 ζh − 4sin 2 (ζh/2) ] +1. (3.3.17)This expression determines the range of values for the CFL number λ for which|g| ≤1. Ultimately, the stability of the Lax-Wendroff scheme for the one-waywave equation is determined.

AMATH 581 ( c○J. N. Kutz) 74Leap-Frog (2,2) for One-Way Wave EquationThe leap-frog discretization for the one-way wave equation yields the iterationscheme()u (m+1)n = u (m−1)n + λ u (m)n+1 − u(m) n−1 . (3.3.18)Plugging in the ansatz (3.3.3) gives the following relations:g m+1 exp(iξ n h)=g m−1 exp(iξ n h)+λg m (exp(iξ n+1 h) − exp(iξ n−1 h))g 2 =1+λg [exp(ih(ξ n+1 − ξ n )) − exp(ih(ξ n−1 − ξ n ))] . (3.3.19)Letting ξ n = nζ reduces the equations further tog 2 =1+λg[exp(iζh) − exp(−iζh)]g − 1 g=2iλ sin(ζh) . (3.3.20)From this we can deduce that|g(ζ)| = √ √g ∗ 1(ζ)g(ζ) =1+λ 2 sin 2 ζh . (3.3.21)Thus to assure stability, it can be shown that we requireλ ≤ 1 . (3.3.22)Thus the leap-frog (2,2) time-stepping scheme is stable for values of λ ≤ 1. Theimplementation of this scheme with this restriction will not forcethesolutionto infinity.AcoupleofgeneralremarksshouldbemadeconcerningthevonNeumannanalysis.• It is a general result that a scheme which is forward in time andcenteredinspace is unstable for the one-way wave equation. This assumes astandardforward discretization, not something like Runge-Kutta.• von Neumann analysis is rarely enough to guarantee stability, i.e. it isnecessary but not sufficient.• Many other mechanisms for unstable growth are not captured by vonNeumann analysis.• Nonlinearity usually kills the von Neumann analysis immediately. Thusalargevarietyofnonlinearpartialdifferentialequationsare beyond thescope of a von Neumann analysis.• Accuracy versus stability: it is always better to worry about accuracy. Anunstable scheme will quickly become apparent by causing the solution toblow-up to infinity. Whereas an inaccurate scheme will simply giveyouawrong result without indicating a problem.

AMATH 581 ( c○J. N. Kutz) 753.4 Comparison of time-stepping schemesAfewopenquestionsremainconcerningthestabilityandaccuracy issues ofthe time- and space-stepping schemes developed. In particular, it is not clearhow the stability results derived in the previous section apply to other partialdifferential equations.Consider, for instance, the leap-frog (2,2) method applied to the one-waywave equation∂u∂t = c∂u ∂x . (3.4.1)The leap-frog discretization for the one-way wave equation yielded the iterationschemeu (m+1)n= u (m−1)n+ λ()u (m)n+1 − u(m) n−1, (3.4.2)where λ = c∆t/∆x is the CFL number. The von Neumann stability analysisbased upon the ansatz u (m)n = g m exp(iξ n h)resultsintheexpressiong − 1 g=2iλ sin(ζh) , (3.4.3)where ξ n = nζ. Thus the scheme was stable provided λ ≤ 1. Note that todouble the accuracy, both the time and space discretizations ∆t and ∆x needto be simultaneously halved.Diffusion EquationWe will again consider the leap frog (2,2) discretization applied to the diffusionequation. The stability properties will be found to be very different than thoseof the one-way wave equation. The difference in the one-way wave equation anddiffusion equation is an extra x derivative so thatDiscretizing the spatial second derivative yields∂u (m)n∂t∂u∂t = uc∂2 ∂x 2 . (3.4.4)= c (∆x 2 u (m)n+1 − 2u(m) n)+ u (m)n−1 . (3.4.5)Second order center differencing in time then yields()u n(m+1) = u n (m−1) +2λ u (m)n+1 − 2u(m) n + u (m)n−1 , (3.4.6)where now the CFL number is given byλ = c∆t∆x 2 . (3.4.7)

AMATH 581 ( c○J. N. Kutz) 76In this case, to double the accuracy and hold the CFL number constant requirescutting the time-step by a factor of four. This is generally true of any partialdifferential equation with the highest derivative term having two derivatives.The von Neumann stability analysis based upon the ansatz u (m)n = g m exp(iξ n h)results in the expressiong m+1 exp(iξ n h)=g m−1 exp(iξ n h)+2λg m (exp(iξ n+1 h) − 2exp(iξ n h)+exp(iξ n−1 h))g − 1 gg − 1 g=2λ(exp(iζh)+exp(−iζh) − 2)=2λ(i sin(ζh) − 1) , (3.4.8)where we let ξ n = nζ. Unlike the one-way wave equation, the addition of theterm −2u (m)n in the discretization makes |g(ζ)| ≥1forallvaluesofλ. Thustheleap-frog (2,2) is unstable for all CFL numbers for the diffusion equation.Interestingly enough, the forward Euler method which was unstable whenapplied to the one-way wave equation can be stable for the diffusion equation.Using an Euler discretization instead of a center difference scheme in time yields()u (m+1)n = u n(m) + λ u (m)n+1 − 2u(m) n + u (m)n−1 . (3.4.9)The von Neumann stability analysis based upon the ansatz u (m)n = g m exp(iξ n h)results in the expressiong =1+λ(i sin(ζh) − 1) , (3.4.10)This givesso that|g| =1− 2λ + λ 2 (1 + sin 2 ζh) , (3.4.11)|g| ≤1 for λ ≤ 1 2 . (3.4.12)Thus the maximum step size in time is given by ∆t =∆x 2 /2c. Again,itisclearthat to double accuracy, the time-step must be reduced by a factor of four.Hyper-DiffusionHigher derivatives in x require central differencing schemes which successivelyadd powers of ∆x to the denominator. This makes for severe time-steppingrestrictions in the CFL number. As a simple example, consider thefourth-orderdiffusion equation∂u∂t = u−c∂4 ∂x 4 . (3.4.13)

AMATH 581 ( c○J. N. Kutz) 77Using a forward Euler method in time and a central-difference scheme in spacegives the iteration scheme()u (m+1)n = u (m)n − λ u (m)n+2 − 4u(m) n+1 +6u(m) n − 4u (m)n−1 + u(m) n−2 , (3.4.14)where the CFL number λ is now given byλ = c∆t∆x 4 . (3.4.15)Thus doubling the accuracy requires a drop in the step-size to ∆t/16, i.e. therun time takes 32 times as long since there are twice as many spatial pointsand 16 times as many time-steps. This kind of behavior is often referredtoasnumerical stiffness.Numerical stiffness of this sort is not a result of the central differencingscheme. Rather, it is an inherent problem with hyper diffusion. For instance,we could consider solving the fourth-order diffusion equation (3.4.13) with FastFourier Transforms. Transforming the equation in the x direction gives∂û∂t = −c(ik)4 û = −ck 4 û. (3.4.16)This can then be solved with a differential equation time-stepping routine. Thetime-step of any of the standard time-stepping routines is based upon the sizeof the right hand side of the equation. If there are n =128Fouriermodes,thenk max =64. Butnotethat(k max ) 4 =(64) 4 =16.8 × 10 6 , (3.4.17)which is a very large value. The time-stepping algorithm will havetoadjusttothese large values which are generated strictly from the physical effect of thehigher-order diffusion.There are a few key issues in dealing with numerical stiffness.• Use a variable time-stepping routine which uses smaller steps when necessarybut large steps if possible. Every built-in MATLAB differentialequation solver uses an adaptive stepping routine to advance thesolution.• If numerical stiffness is a problem, then it is often useful to use an implicitscheme. This will generally allow for larger time-steps. The time-steppingalgorithm ode113 uses a predictor-corrector method which partially utilizesan implicit scheme.• If stiffness comes from the behavior of the solution itself, i.e. you areconsidering a singular problem, then it is advantageous to use a solverspecifically built for this stiffness. The time-stepping algorithm ode15srelies on Gear methods and is well suited to this type of problem.• From a practical viewpoint, beware of the accuracy of ode23 or ode45 whenstrong nonlinearity or singular behavior is expected.

AMATH 581 ( c○J. N. Kutz) 783.5 Optimizing computational performance: rules of thumbComputational performance is always crucial in choosing a numerical scheme.Speed, accuracy and stability all play key roles in determining the appropriatechoice of method for solving. We will consider three prototypical equations:∂u∂t = ∂u∂xone-way wave equation (3.5.1a)∂u∂t = ∂2 u∂x 2 diffusion equation (3.5.1b)i ∂u∂t = 1 ∂ 2 u2 ∂x+ |u| 2 u 2 nonlinear Schrödinger equation . (3.5.1c)Periodic boundary conditions will be assumed in each case. The purpose of thissection is to build a numerical routine which will solve these problemsusingtheiteration procedures outlined in the previous two sections. Of specific interestwill be the setting of the CFL number and the consequences of violating thestability criteria associated with it.The equations are considered in one dimension such that the first and secondderivative are given by⎡⎤0 1 0 ··· 0 −1⎢ −1 0 1 0 ··· 0 ⎥∂u∂x → 12∆x⎢⎣0. .. . .. . ....0.. ··· 0 −1 0 11 0 ··· 0 −1 0⎛⎜⎝⎥⎦u 1u 2.u n⎞⎟⎠(3.5.2)and⎡∂ 2 u∂x 2 → 1∆x 2 ⎢⎣⎤−2 1 0 ··· 0 11 −2 1 0 ··· 0⎛.0 .. . .. . ....⎜⎝0⎥⎦. ··· 0 1 −2 11 0 ··· 0 1 −2u 1u 2..u n⎞⎟⎠ . (3.5.3)From the previous lectures, we have the following discretization schemes forthe one-way wave equation (3.5.1):Euler (unstable):u (m+1)n =u (m)n + λ (2leap-frog (2,2) (stable for λ ≤ 1): u n(m+1) =u (m−1)n +λu (m)n+1 −u(m) n−1(u (m)n+1 −u(m) n−1)(3.5.4a))(3.5.4b)

AMATH 581 ( c○J. N. Kutz) 79Figure 20: Evolution of the one-way wave equation with the leap-frog (2,2)scheme and with CFL=0.5. The stable traveling wave solution is propagated inthis case to the left.where the CFL number is given by λ =∆t/∆x.equation (3.5.1)(Euler (stable for λ ≤ 1/2): u (m+1)n =u n (m) +λleap-frog (2,2) (unstable):u (m+1)n =u (m−1)n +2λSimilarly for the diffusionu (m)n+1 −2u(m) n(u (m))+u(m) n−1n+1 −2u(m) n+u(m) n−1(3.5.5a))(3.5.5b)where now the CFL number is given by λ =∆t/∆x 2 .ThenonlinearSchrödingerequation discretizes to the following form:∂u (m)n∂t= − i (2∆x 2 u (m)n+1 − 2u(m) n)+ u (m)n−1− i|u n(m) | 2 u (m)n . (3.5.6)We will explore Euler and leap-frog (2,2) time-stepping withthisequation.One-Way Wave EquationWe first consider the leap-frog (2,2) scheme applied to the one-way wave equation.Figure 20 depicts the evolution of an initial Gaussian pulse. For this case,

AMATH 581 ( c○J. N. Kutz) 80Figure 21: Evolution of the one-way wave equation using the leap-frog (2,2)scheme with CFL=2 (left) along with the Euler time-stepping scheme (right).The analysis predicts stable evolution of leap-frog provided the CFL≤ 1. Thusthe onset of numerical instability near t ≈ 3fortheCFL=2caseisnotsurprising.Likewise, the Euler scheme is expected to be unstable for all CFL.the CFL=0.5 so that stable evolution is analytically predicted. The solutionpropagates to the left as expected from the exact solution. The leap-frog (2,2)scheme becomes unstable for λ ≥ 1andthesystemisalwaysunstablefortheEuler time-stepping scheme. Figure 21 depicts the unstable evolution of theleap-frog (2,2) scheme with CFL=2 and the Euler time-stepping scheme. Theinitial conditions used are identical to that in Fig. 20. Since we have predictedthat the leap-frog numerical scheme is only stable provided λ

AMATH 581 ( c○J. N. Kutz) 81% initial conditionsu0=exp(-x.^2)’;u1=exp(-(x+dt).^2)’;usol(:,1)=u0;usol(:,2)=u1;% sparse matrix for derivative terme1=ones(n,1);A=spdiags([-e1 e1],[-1 1],n,n);A(1,n)=-1; A(n,1)=1;% leap frog (2,2) or euler iteration schemefor j=1:time_steps-1% u2 = u0 + CFL*A*u1; % leap frog (2,2)% u0 = u1; u1 = u2; % leap frog (2,2)u2 = u1 + 0.5*CFL*A*u1; % euleru1 = u2;% eulerusol(:,j+2)=u2;end% plot the datawaterfall(x,t,usol’);map=[0 0 0];colormap(map);% set x and y limits and fontsizeset(gca,’Xlim’,[-L/2 L/2],’Xtick’,[-L/2 0 L/2],’FontSize’,[20]);set(gca,’Ylim’,[0 Time],’ytick’,[0 Time/2 Time],’FontSize’,[20]);view(25,40)% set axis labels and fontsxl=xlabel(’x’); yl=ylabel(’t’); zl=zlabel(’u’);set(xl,’FontSize’,[20]);set(yl,’FontSize’,[20]);set(zl,’FontSize’,[20]);print -djpeg -r0 fig.jpg % print jpeg at screen resolutionHeat EquationIn a similar fashion, we investigate the evolution of the diffusion equation whenthe space-time discretization is given by the leap-frog (2,2) scheme or Eulerstepping. Figure 22 shows the expected diffusion behavior for thestableEulerscheme (λ ≤ 0.5). In contrast, Fig. 23 shows the numerical instabilities which are

AMATH 581 ( c○J. N. Kutz) 82Figure 22: Stable evolution of the heat equation with the Euler scheme withCFL=0.5. The initial Gaussian is diffused in this case.generated from violating the CFL constraint for the Euler scheme or using thealways unstable leap-frog (2,2) scheme for the diffusion equation. The numericalcode used to generate these solutions follows that given previously for the onewaywave equation. However, the sparse matrix is now given by% sparse matrix for second derivative terme1=ones(n,1);A=spdiags([e1 -2*e1 e1],[-1 0 1],n,n);A(1,n)=1; A(n,1)=1;Further, the iterative process is now% leap frog (2,2) or euler iteration schemefor j=1:time_steps-1u2 = u0 + 2*CFL*A*u1; % leap frog (2,2)u0 = u1; u1 = u2; % leap frog (2,2)% u2 = u1 + CFL*A*u1; % euler% u1 = u2; % eulerusol(:,j+2)=u2;end

AMATH 581 ( c○J. N. Kutz) 83Figure 23: Evolution of the heat equation with the Euler time-stepping scheme(left) and leap-frog (2,2) scheme (right) with CFL=1. The analysis predictsthat both these schemes are unstable. Thus the onset of numerical instabilityis observed.where we recall that the CFL condition is now given by λ =∆t/∆x 2 ,i.e.CFL=dt/dx/dxThis solves the one-dimensional heat equation with periodic boundaryconditions.Nonlinear Schrödinger EquationThe nonlinear Schrödinger equation can easily be discretized by the above techniques.However, as with most nonlinear equations, it is a bit moredifficulttoperform a von Neumann analysis. Therefore, we explore the behavior for thissystem for two different discretization schemes: Euler and leap-frog (2,2). TheCFL number will be the same with both schemes (λ =0.05) and the stabilitywill be investigated through numerical computations. Figure 24 shows the evolutionof the exact one-soliton solution of the nonlinear Schrödinger equation(u(x, 0) = sech(x)) over six units of time. The Euler scheme is observed tolead to numerical instability whereas the leap-frog (2,2) scheme is stable. Ingeneral, the leap-frog schemes work well for wave propagation problems whileEuler methods are better for problems of a diffusive nature.The MATLAB code modifications necessary to solve the nonlinear Schrödingerequation are trivial. Specifically, the iteration scheme requires change. For thestable leap-frog scheme, the following command structure is requiredu2 = u0 + -i*CFL*A*u1- i*2*dt*(conj(u1).*u1).*u1;u0 = u1; u1 = u2;

AMATH 581 ( c○J. N. Kutz) 84Figure 24: Evolution of the nonlinear Schrödinger equation withtheEulertimesteppingscheme (left) and leap-frog (2,2) scheme (right) with CFL=0.05.Note that i is automatically defined in MATLAB as i = √ −1. Thus it isimperative that you do not use the variable i as a counter in your FOR loops.You will solve a very different equation if not careful with this definition.4 Spectral MethodsSpectral methods are one of the most powerful solution techniques for ordinaryand partial differential equations. The best known example ofaspectralmethodis the Fourier transform. We have already made use of the Fourier transformusing FFT routines. Other spectral techniques exist which render a variety ofproblems easily tractable and often at significant computational savings.4.1 Fast-Fourier Transforms and Cosine/Sine transformFrom our previous lectures, we are already familiar with the Fourier transformand some of its properties. At the time, the distinct advantage to using theFFT was its computational efficiency of solving a problem in O(N log N). Thislecture explores the underlying mathematical reasons for such performance.One of the abstract definitions of a Fourier transform pair is given byF (k) =∫ ∞f(x) = 12π−∞∫ ∞e −ikx f(x)dx−∞e ikx F (k)dk .(4.1.1a)(4.1.1b)On a practical level, the value of the Fourier transform revolves squarely aroundthe derivative relationship̂f (n) =(ik) n ̂f (4.1.2)

AMATH 581 ( c○J. N. Kutz) 85which results from the definition of the Fourier transform and integrationbyparts. Recall that we denote the Fourier transform of f(x) asf(x).̂When considering a computational domain, the solution can only be foundon a finite length domain. Thus the definition of the Fourier transform needsto be modified in order to account for the finite sized computational domain.Instead of expanding in terms of a continuous integral for values of wavenumberk and cosines and sines (exp(ikx)), we expand in a Fourier seriesF (k) =N∑n=1f(n) = 1 N[f(n)exp −iN∑k=12π(k − 1)N](n − 1)[]2π(k − 1)F (k)exp i (n − 1)N1 ≤ k ≤ N (4.1.3a)1 ≤ n ≤ N.(4.1.3b)Thus the Fourier transform is nothing but an expansion in a basis of cosine andsine functions. If we define the fundamental oscillatory piece as( ) ( ) ( )2iπ(k−1)(n−1) 2π(k−1)(n−1) 2π(k−1)(n−1)w nk =exp=cos+ i sin,NNN(4.1.4)then the Fourier transform results in the expressionF n =N−1∑k=0w nk f k 0 ≤ n ≤ N − 1 . (4.1.5)Thus the calculation of the Fourier transform involves a double sum and anO(N 2 )operation. Thus,atfirst,itwouldappearthattheFouriertransformmethod is the same operation count as LU decomposition. The basis functionsused for the Fourier transform, sine transform and cosine transform are depictedin Fig. 25. The process of solving a differential or partial differential equationinvolves evaluating the coefficient of each of the modes. Note that this expansion,unlike the finite difference method, is a global expansion in that every basisfunction is evaluated on the entire domain.The Fourier, sine, and cosine transforms behave very differently at theboundaries. Specifically, the Fourier transform assumes periodic boundary conditionswhereas the sine and cosine transforms assume pinnedandno-fluxboundariesrespectively. The cosine and sine transform are often chosen for theirboundary properties. Thus for a given problem, an evaluation mustbemadeof the type of transform to be used based upon the boundary conditions needingto be satisfied. Table 9 illustrates the three different expansions and theirassociated boundary conditions. The appropriate MATLAB command is alsogiven.

AMATH 581 ( c○J. N. Kutz) 8610.50−0.5−10 π 2π10.50−0.5−10 π 2π10.50−0.5mode 1mode 2mode 3mode 4mode 5−10 π 2πFigure 25: Basis functions used for a Fourier mode expansion (top), a sineexpansion (middle), and a cosine expansion (bottom).Fast-Fourier Transforms: Cooley-Tukey AlgorithmTo see how the FFT gets around the computational restriction of an O(N 2 )scheme, we consider the Cooley-Tukey algorithm which drops the computationsto O(N log N). To consider the algorithm, we begin with the N × N matrix F Nwhose components are given by(F N ) jk = w jkn =exp(i2πjk/N) (4.1.6)The coefficients of the matrix are points on the unit circle since |wn jk | =1. Theyare also the basis functions for the Fourier transform.The FFT starts with the following trivial observationw 2 2n = w n , (4.1.7)

AMATH 581 ( c○J. N. Kutz) 87command expansion boundary conditionsfft F k = ∑ 2N−1j=0f j exp(iπjk/N) periodic: f(0) = f(L)dst F k = ∑ N−1j=1 f j sin(πjk/N) pinned: f(0) = f(L) =0dct F k = ∑ N−2j=0 f j cos(πjk/2N) no-flux: f ′ (0) = f ′ (L) =0Table 9: MATLAB functions for Fourier, sine, and cosine transforms and theirassociated boundary conditions. To invert the expansions, the MATLAB commandsare ifft, idst, and idct repectively.which is easy to show since w n =exp(i2π/n) andw 2 2n =exp(i2π/(2n)) exp(i2π/(2n)) = exp(i2π/n) =w n . (4.1.8)The consequences to this simple relationship are enormous and at the core ofthe success of the FFT algorithm. Essentially, the FFT is a matrix operationy = F N x (4.1.9)which can now be split into two separate operations. Thus defining⎛ ⎞⎛ ⎞x 0x 1x 2x 3x e =x 4and x o =x 5⎜⎝⎟⎜. ⎠⎝⎟. ⎠x N−2x N−1(4.1.10)which are both vectors of length M = N/2, we can form the two M ×M systemsy e = F M x e and y o = F M x o (4.1.11)for the even coefficient terms x e , y e and odd coefficient terms x o , y o .Thusthecomputation size goes from O(N 2 )toO(2M 2 )=O(N 2 /2). However, we mustbe able to reconstruct the original y from the smaller system of y e and y o .Andindeed we can reconstruct the original y. In particular, it can be shown thatcomponent by componenty n = y e n + w n N y o n n =0, 1, 2,...,M − 1 (4.1.12a)y n+M = y e n − wn N yo n n =0, 1, 2,...,M − 1 . (4.1.12b)This is where the shift occurs in the FFT routine which maps the domainx ∈ [0,L]to[−L, 0] and x ∈ [−L, 0] to [0,L]. The command fftshift undoes thisshift. Details of this construction can be found elsewhere [7, 11].

AMATH 581 ( c○J. N. Kutz) 88x 0x 1x 2x 3N=2 3ey 0ey 1ey 2ey 3y 0y 1y 2y 3x 4y 0oy 4x 5y 1oy 5x 6x 7yoy 2y 6oy 37Figure 26: Graphical description of the Fast Fourier Transform process whichsystematically continues to factor the problem in two. This process allows theFFT routine to drop to O(N log N) operations.There is no reason to stop the splitting process at this point. Infact,providedwe choose the size of our domain and matrix F N so that N is a power of two,then we can continue to split the system until we have a simple algebraic, i.e.a1× 1system,solutiontoperform. Theprocessisillustratedgraphically inFig. 26 where the switching and factorization are illustrated. Once the finallevel is reached, the algebraic expression is solved and the process is reversed.This factorization process renders the FFT scheme O(N log N).4.2 Chebychev Polynomials and TransformThe Fast Fourier Transform is only one of many possible expansion bases, i.e.there is nothing special about expanding in cosines and sines. Of course, theFFT expansion does have the unusual property of factorization which drops it

AMATH 581 ( c○J. N. Kutz) 89to an O(N log N) scheme. Regardless,thereareamyriadofotherexpansionbases which can be considered. The primary motivation for considering otherexpansions is based upon the specifics of the given governing equations and itsphysical boundaries and constraints. Special functions are oftenprimecandidatesfor use as expansion bases. The following are some important examples• Bessel functions: radial, 2D problems• Legendre polynomials: 3D Laplaces equation• Hermite-Gauss polynomials: Schrödinger with harmonic potential• Spherical harmonics: radial, 3D problems• Chebychev polynomials: bounded 1D domainsThe two key properties required to use any expansion basis successfully are itsorthogonality properties and the calculation of the norm. Regardless, all theabove expansions appear to require O(N 2 )calculations.Chebychev PolynomialsAlthough not often encountered in mathematics courses, the Chebychev polynomialis an important special function from a computational viewpoint. Thereason for its prominence in the computational setting will become apparentmomentarily. For the present, we note that the Chebychev polynomials aresolutions to the differential equation√1 − x2 ddx( √1− x2 dT ndx)+ n 2 T n =0 x ∈ [−1, 1] . (4.2.1)This is a self-adjoint Sturm-Lioville problem. Thus the following properties areknown1. eigenvalues are real: λ n = n 22. eigenfunctions are real: T n (x)3. eigenfunctions are orthogonal:∫ 1−1(1 − x 2 ) −1/2 T n (x)T m (x)dx = π 2 c nδ nm (4.2.2)where c 0 =2,c n =1(n>0) and δ nm is the Delta function4. eigenfunctions form a complete basis

AMATH 581 ( c○J. N. Kutz) 901.0T n T 00.5T 1T 2T 3T 40.0−0.5−1.0−1 −0.5 0 0.5 1xFigure 27: The first five Chebychev polynomials over the the interval of definitionx ∈ [−1, 1].Each Chebychev polynomial (of degree n) isdefinedbyThus we findT n (cos θ) =cosnθ . (4.2.3)T 0 (x) =1T 1 (x) =xT 2 (x) =2x 2 − 1T 3 (x) =4x 3 − 3xT 4 (x) =8x 4 − 8x 2 +1.(4.2.4a)(4.2.4b)(4.2.4c)(4.2.4d)(4.2.4e)The behavior of the first five Chebychev polynomials is illustrated in Fig. 27.It is appropriate to ask why the Chebychev polynomials, of all thespecialfunctions listed, are of such computational interest. Especially given that theequation which the T n (x) satisfy,andtheirfunctionalformshowninFig.27,appearto be no better than Bessel, Hermite-Gauss, or any other special function.

AMATH 581 ( c○J. N. Kutz) 91The distinction with the Chebychev polynomials is that you can transform themso that use can be made of the O(N log N) discretecosinetransform. Thiseffectivelyyields the Chebychev expansion scheme an O(N log N) transformation.Specifically, we transform from the interval x ∈ [−1, 1] by lettingx =cosθ θ ∈ [0,π] . (4.2.5)Thus when considering a function f(x), we have f(cos θ) =g(θ). Under differentiationwe finddgdθ = −f ′ · sin θ (4.2.6)Thus dg/dθ =0atθ =0,π,i.e.no-fluxboundaryconditionsaresatisfied.Thisallows us to use the dct (discrete cosine transform) to solve a given problem inthe new transformed variables.The Chebychev expansion is thus given byf(x) =∞∑a k T k (x) (4.2.7)k=0where the coefficients a k are determined from orthogonality and inner productsto be∫ 11a k = √ f(x)T k(x)dx . (4.2.8)1 − x2−1It is these coefficients which are calculated in O(N log N) time. Someoftheproperties of the Chebychev polynomials are as follows:• T n+1 =2xT n (x) − T n−1 (x)•|T n (x)| ≤1, |T ′ n(x)| ≤n 2• T n (±1) = (±1) n• d p /dx p (T n (±1)) = (±1) n+p ∏ p−1k=0 (n2 − k 2 )/(2k +1)• if n is even (odd), T n (x) iseven(odd)There are a couple of critical practical issues which must be considered whenusing the Chebychev scheme. Specifically the grid generation andspatialresolutionare a little more difficult to handle. In using the discrete cosine transformon the variable θ ∈ [0,π], we recall that our original variable is actually x =cosθwhere x ∈ [−1, 1]. Thus the discretization of the θ variable leads to( )(2m − 1)πx m =cosm =1, 2,...,n. (4.2.9)2nThus although the grid points are uniformly spaced in θ, thegridpointsareclustered in the original x variable. Specifically, there is a clustering of grid

AMATH 581 ( c○J. N. Kutz) 921x m=cos[(2m−1)π/2n]m=1,2,...,n0−1−1 −0.5 0 0.5 1xFigure 28: Clustered grid generation for n =30pointsusingtheChebychevpolynomials. Note that although the points are uniformaly spaced in θ, theyare clustered due to the fact that x m =cos[(2m − 1)π/2n] wherem =1, 2,...,n.points at the boundaries. The Chebychev scheme then automatically has higherresolution at the boundaries of the computational domain. The clustering ofthe grid points at the boundary is illustrated in Fig. 28. So as theresolutionisincreased, it is important to be aware that the resolution increase is not uniformacross the computational domain.Solving Differential EquationsAs with any other solution method, a solution scheme must haveanefficientwayof relating derivatives to the function itself. For the FFT method, there was avery convenient relationship between the transform of a function and the transformof its derivatives. Although not as transparent as the the FFT method, wecan also relate the Chebychev transform derivatives to the Chebychev transformitself.Defining L to be a linear operator so thatLf(x) =∞∑b n T n (x) , (4.2.10)n=0then with f(x) = ∑ ∞n=0 a nT n (x) wefind• Lf = f ′ (x) :c n b n =2 ∑ ∞p=n+1(p+nodd) pa p• Lf = f ′′ (x) :c n b n = ∑ ∞p=n+2(p+neven)• Lf = xf(x) :b n =(c n−1 a n−1 + a n+1 )/2

AMATH 581 ( c○J. N. Kutz) 93• Lf = x 2 f(x) :b n =(c n−2 a n−2 +(c n + c n−1 )a n + a n+2 )/4where c 0 =2,c n =0(n0),d n =1(n ≥ 0), and d n =0(n

AMATH 581 ( c○J. N. Kutz) 94• Fourier transform FFT(u 3 )3. u 3 d 2 u/dx 2• determine d 2 u/dx 2 →• multiply by u 3 → u 3 d 2 u/dx 2• Fourier transform FFT(u 3 d 2 u/dx 2 )̂ d 2 u/dx 2 =(ik) 2 û, d 2 u/dx 2 = FFT −1 (−k 2 û)These examples give an outline of how the nonlinear, non-constant coefficientschemes would work in practice.To illustrate the implementation of these ideas, we solve the advectiondiffusionequations spectrally. Thus far the equations have been consideredlargely with finite difference techniques. However, the MATLAB codes presentedhere solve the equations using the FFT for both the streamfunction andvorticity evolution. We begin by initializing the appropriate numerical parametersclear all; close all;nu=0.001;Lx=20; Ly=20; nx=64; ny=64; N=nx*ny;x2=linspace(-Lx/2,Lx/2,nx+1); x=x2(1:nx);y2=linspace(-Ly/2,Ly/2,ny+1); y=y2(1:ny);Thus the computational domain is for x ∈ [−10, 10] and y ∈ [−10, 10]. Thediffusion parameter is chosen to be ν =0.001. The initial conditions are thendefined as a stretched Gaussian% INITIAL CONDITIONS[X,Y]=meshgrid(x,y);w=1*exp(-0.25*X.^2-Y.^2);figure(1), pcolor(abs(w)); shading interp; colorbar; drawnowThe next step is to define the spectral k values in both the x and y directions.This allows us to solve for the streamfunction (2.6.8) spectrally. Oncethe streamfunction is determined, a call is made to a time-stepping schemewhich steps the vorticity solution forward in time. In this example, the streamfunctionis updated every ∆t =0.5. The loop is run through ten times for atotal time of t =5.% SPECTRAL K VALUESkx=(2*pi/Lx)*[0:(nx/2-1) (-nx/2):-1]’; kx(1)=10^(-6);ky=(2*pi/Ly)*[0:(ny/2-1) (-ny/2):-1]’; ky(1)=10^(-6);

AMATH 581 ( c○J. N. Kutz) 95% Streamfunctionfor stream_loop=1:10wt=fft2(w)/N;for j=1:nypsit(:,j)=-wt(:,j)./(kx.^2+ky(j)^2);endfor j=1:nxpsitx(j,:)=i*(psit(j,:).*kx’);endpsix=real(ifft2(psitx*N));for j=1:nypsity(:,j)=i*(psit(:,j).*ky);endpsiy=real(ifft2(psity*N));wt2=reshape(wt,N,1);[t,wsol]=ode23(’spc_rhs’,[0 0.5],wt2,[],psix,psiy,kx,ky,nu,nx,ny,N);wt2=wsol(end,:)’;wt=reshape(wt2,nx,ny);w=ifft2(wt*N);figure(1)pcolor(abs(w)); shading interp; colorbar; drawnowendThe right hand side of the system of differential equations which results fromFourier transforming is contained within the function spc rhs.m. Theoutlineofthis routine is provided below. Aside from the first function call line, the restof the routine first finds the derivatives of the vorticity with respecttox and yrespectively. The derivatives of the streamfunction are sent into the program.Once these terms are determined, the matrix components are reshaped and alarge system of differential equations is solved.function rhs=spc_rhs(tspan,wt2,dummy,psix,psiy,kx,ky,nu,nx,ny,N);wt=reshape(wt2,nx,ny);% w_xfor j=1:nxwtx(j,:)=i*(wt(j,:).*kx’);endwx=ifft2(wtx*N);

AMATH 581 ( c○J. N. Kutz) 96% w_yfor j=1:nywty(:,j)=i*(wt(:,j).*ky);endwy=ifft2(wty*N);% transform w_x*psi_y and w_y*psi_x and reshapewxpyt=fft2(wx.*psiy)/N;wypxt=fft2(wy.*psix)/N;wxpyt2=reshape(wxpyt,N,1);wypxt2=reshape(wypxt,N,1);% Laplacian Termsfor j=1:nxwtxx(j,:)=-wt(j,:).*(kx.^2)’;endfor j=1:nywtyy(:,j)=-wt(:,j).*(ky.^2);endwtxx2=reshape(wtxx,N,1);wtyy2=reshape(wtyy,N,1);rhs=(nu*(wtxx2+wtyy2)-wypxt2+wxpyt2);The code will quickly and efficiently solve the advection-diffusion equations intwo dimensions. Figure 14 demonstrates the evolution of the initial stretchedGaussian vortex over t ∈ [0, 8].4.4 Pseudo-spectral techniques with filteringThe decomposition of the solution into Fourier mode components does not alwayslead to high performance computing. Specifically when some form of numericalstiffness is present, computational performance can sufferdramatically.However, there are methods available which can effectively eliminate some ofthe numerical stiffness by making use of analytic insight into theproblem.We consider again the example of hyper-diffusion for which thegoverningequations are∂u∂t = − ∂4 u∂x 4 . (4.4.1)In the Fourier domain, this becomesdûdt = −(ik)4 û = −k 4 û (4.4.2)

AMATH 581 ( c○J. N. Kutz) 97which has the solutionû = û 0 exp(−k 4 t) . (4.4.3)However, a time-stepping scheme would obviously solve (4.4.2) directly withoutmaking use of the analytic solution.For n =128Fouriermodes,thewavenumberk ranges in values from k ∈[−64, 64]. Thus the largest value of k for the hyper-diffusion equation isk 4 max =(64)4 =16, 777, 216 , (4.4.4)or roughly k 4 max =1.7 × 107 .Forn =1024Fouriermodes,thisisk 4 max =6.8 × 1010 . (4.4.5)Thus even if our solution is small at the high wavenumbers, this can createproblems. For instance, if the solution at high wavenumbers is O(10 −6 ), thendûdt = −(1010 )(10 −6 )=−10 4 . (4.4.6)Such large numbers on the right hand side of the equations forces time-steppingschemes like ode23 and ode45 to take much smaller time-steps in order to maintaintolerance constraints. This is a form of numerical stiffness which can becircumvented.Filtered Pseudo-SpectralThere are a couple of ways to help get around the above mentioned numericalstiffness. We again consider the very general partial differential equation∂u∂t= Lu + N(u) (4.4.7)where as before L is a linear, constant coefficient operator, i.e. it can take theform L = ad 2 /dx 2 + bd/dx + c where a, b, and c are constants. The second termN(u) includesthenonlinearandnon-constantcoefficientterms.An example ofthis would be N(u) =u 3 + f(x)u + g(x)d 2 u/dx 2 .Previously, we transformed the equation by Fourier transforming and constructinga system of differential equations. However, there isabetterwaytohandle this general equation and remove some numerical stiffness at the sametime. To introduce the technique, we consider the first-order differentialequationdy+ p(t)y = g(t) . (4.4.8)dtWe can multiply by the integrating factor µ(t) sothatµ dy + µp(t)y = µg(t) . (4.4.9)dt

AMATH 581 ( c○J. N. Kutz) 98We note from the chain rule that (µy) ′ = µ ′ y + µy ′ where the prime denotesdifferentiation with respect to t. Thusthedifferentialequationbecomesd(µy) =µg(t) , (4.4.10)dtprovided dµ/dt = µp(t). The solution then becomesy = 1 [∫](∫µ(t)g(t)dt + c µ(t) =expµ)p(t)dt(4.4.11)which is the standard integrating factor method of a first course on differentialequations.We use the key ideas of the integrating factor method to help solve thegeneral partial differential equation and remove stiffness. Fourier transforming(4.4.7) results in the spectral system of differential equationsThis can be rewrittenMultiplying by exp(−α(k)t) givesdûdt = α(k)û + ̂ N(u) . (4.4.12)dûdt − α(k)û = ̂ N(u) . (4.4.13)dûexp(−α(k)t) − α(k)û exp(−α(k)t) =exp(−α(k)t) N(u) ̂dtd[û exp(−α(k)t)] = exp(−α(k)t) N(u) ̂ .dtBy defining ̂v = û exp(−α(k)t), the system of equations reduces tod̂vdt =exp(−α(k)t) N(u) ̂û = ̂v exp(α(k)t) .(4.4.14a)(4.4.14b)Thus the linear, constant coefficient terms are solved for explicitly, and thenumerical stiffness associated with the Lu term is effectively eliminated.Example: Fisher-Kolmogorov EquationAs an example of the implementation of the the filtered pseudo-spectral scheme,we consider the Fisher-Kolmogorov equation∂u∂t = ∂2 u∂x 2 + u3 + cu . (4.4.15)

AMATH 581 ( c○J. N. Kutz) 99Fourier transforming the equation yieldswhich can be rewrittenThus α(k) =c − k 2 anddûdt =(ik)2 û + û3 + cû (4.4.16)dûdt +(k2 − c)û = û3 . (4.4.17)d̂v=exp[−(c − k2 )t]û3(4.4.18a)dtû = ̂v exp[(c − k 2 )t] .(4.4.18b)It should be noted that the solutions u must continually be updating the valueof u 3 which is being transformed in the right hand side of the equations. Thusafter every time-step ∆t, thenewu should be used to evaluate û3 .There are a couple of practical issues that should be considered when implementingthis technique• When solving the general equation∂u∂tit is important to only step forward ∆t in time with= Lu + N(u) (4.4.19)d̂vdt =exp(−α(k)t) N(u) ̂û = ̂v exp(α(k)t)before the nonlinear term ̂ N(u) isupdated.• The computational savings for this method generally does not manifestitself unless there are more than two spatial derivatives in the highestderivative of Lu.• Care must be taken in handling your time-step ∆t in MATLAB since ituses adaptive time-stepping.Comparison of Spectral and Finite Difference MethodsBefore closing the discussion on the spectral method, we investigate the advantagesand disadvantages associated with the spectral and finitedifferenceschemes. Of particular interest are the issues of accuracy, implementation, computationalefficiency, and boundary conditions. The strengths and weaknessesof the schemes will be discussed.

AMATH 581 ( c○J. N. Kutz) 100A. Accuracy• Finite Differences: Accuracy is determined by the ∆x and ∆ychosen in the discretization. Accuracy is fairly easy to compute andgenerally much worse than spectral methods.• Spectral Method: Spectral methods rely on a global expansion andare often called spectrally accurate. In particular, spectral methodshave infinite order accuracy. Althoughthedetailsofwhatthismeanswill not be discussed here, it will suffice to say that they are generallyof much higher accuracy than finite differences.B. Implementation• Finite Differences: The greatest difficulty in implementing thefinite difference schemes is generating the correct sparse matrices.Many of these matrices are very complicated with higher orderschemesand in higher dimensions. Further, when solving the resulting systemAx = b, itshouldalwaysbecheckedwhetherdetA =0. TheMAT-LAB command cond(A) checks the condition number of the matrix.If cond(A)> 10 15 ,thendetA ≈ 0andstepsmustbetakeninorderto solve the problem correctly.• Spectral Method: The difficulty with using FFTs is the continualswitching between the time or space domain and the spectral domain.Thus it is imperative to know exactly when and where in thealgorithm this switching must take place.C. Computational Efficiency• Finite Differences: The computational time for finite differences isdetermined by the size of the matrices and vectors in solving Ax = b.Generally speaking, you can guarantee O(N 2 )efficiencybyusingLUdecomposition. At times, iterative schemes can lower the operationcount, but there are no guarantees about this.• Spectral Method: The FFT algorithm is an O(N log N)operation.Thus it is almost always guaranteed to be faster than the finite differencesolution method which is O(N 2 ). Recall that this efficiencyimprovement comes with an increased accuracy as well. Thus makingthe spectral highly advantageous when implemented.D. Boundary Conditions• Finite Differences: Of the above categories, spectral methods aregenerally better in every regard. However, finite differencesareclearly

AMATH 581 ( c○J. N. Kutz) 101superior when considering boundary conditions. Implementing thegeneric boundary conditionsαu(L)+β du(L)dx = γ (4.4.20)is easily done in the finite difference framework. Also, more complicatedcomputational domains may be considered. Generally anycomputational domain which can be constructed of rectangles iseasilyhandled by finite difference methods.• Spectral Method: Boundary conditions are the critical limitationon using the FFT method. Specifically, only periodic boundary conditionscan be considered. The use of the discrete sine or cosinetransform allows for the consideration of pinned or no-flux boundaryconditions, but only odd or even solutions are admitted respectively.4.5 Boundary conditions and the Chebychev TransformThus far, we have focused on the use of the FFT as the primary tool for spectralmethods and their implementation. However, many problems donotinfacthaveperiodic boundary conditions and the accuracy and speed of the FFT is rendereduseless. The Chebychev polynomials are a set of mathematical functionswhichstill allow for the construction of a spectral method which is bothfastandaccurate. The underlying concept is that Chebychev polynomials can be relatedto sines and cosines, and therefore they can be connected to the FFT routine.Before constructing the details of the Chebychev method, we begin by consideringthree methods for handling non-periodic boundary conditions.Method 1: periodic extension with FFTsSince the FFT only handles periodic boundary conditions, we can periodicallyextend a general function f(x) inordertomakethefunctionitselfnowperiodic.The FFT routine can now be used. However, the periodic extension willin general generate discontinuities in the periodically extended function. Thediscontinuities give rise to Gibb’s phenomena: strong oscillations and errors areaccumulated at the jump locations. This will greatly effect the accuracy of thescheme. So although spectral accuracy and speed is retained away from discontinuities,the errors and the jumps will begin to propagate out to the rest of thecomputational domain.To see this phenomena, Fig. 29a considers a function which is periodicallyextended as shown in Fig. 29b. The FFT approximation to this function isshown in Fig. 29c where the Gibb’s oscillations are clearly seen at the jump locations.The oscillations clearly impact the usefulness of this periodic extensiontechnique.

AMATH 581 ( c○J. N. Kutz) 102yy1.20.80.40-0.4-0.8(a)-1.2-1 -0.5 0 0.5 1x1.20.80.40-0.4-0.8y1.20.80.40-0.4-0.8(b)-1.2-3 -2 -1 0 1 2 3xn=8n=16-1.2-3 -2 -1 0 1 2 3xFigure 29: The function y(x) =x for x ∈ [−1, 1] (a) and its periodic extension(b). The FFT approximation is shown in (c) for n =8Fouriermodesandn =16Fouriermodes. NotetheGibb’soscillations.(c)Method 2: polynomial approximation with equi-spaced pointsIn moving away from an FFT basis expansion method, we can consider the moststraight forward method available: polynomial approximation. Thus we simplydiscretize the given function f(x) withN +1equallyspacedpointsandfitanN th degree polynomial through it. This amounts to lettingf(x) ≈ a 0 + a 1 x + a 2 x 2 + a 3 x 3 + ···+ a n x N (4.5.1)where the coefficients a n are determined by an (N +1)×(N +1) system of equations.This method easily satisfies the prescribed non-periodic boundary conditions.Further, differentiation of such an approximation is trivial. In this casehowever, Runge phenomena (polynomial oscillations) generally occurs. This isbecause a polynomial of degree n generally has N − 1combinedmaximumsandminimums.The Runge phenomena can easily be illustrated with the simple examplefunction f(x) =(1+16x 2 ) −1 .Figure30(a)-(b)illustratesthelargeoscillationswhich develop near the boundaries due to Runge phenomena for N =12andN =24. AswiththeGibb’soscillationsgeneratedbyFFTs,theRunge oscil-

AMATH 581 ( c○J. N. Kutz) 103yy1.210.80.60.40.20−0.21.210.80.60.40.20(a)(c)−0.2−1 −0.5 0 0.5 1x(b)(d)−1 −0.5 0 0.5 1xFigure 30: The function y(x) =(1+16x 2 ) −1 for x ∈ [−1, 1] (dotted line) withthe bold points indicating the grid points for equi-spaced spaced points (a)n =12and(b)n =24andChebychevclusteredpoints(c)n =12and(d)n =24. Thesolidlinesarethepolynomialapproximationsgenerated using theequi-spaced points (a)-(b) and clustered points (c)-(d) respectively.lations render a simple polynomial approximation based upon equallyspacedpoints useless.Method 3: polynomial approximation with clustered pointsThere exists a modification to the straight forward polynomial approximationgiven above. Specifically, this modification constructs a polynomial approximationon a clustered grid as opposed to the equal spacing which led to Rungephenomena. This polynomial approximation involves a clustered grid which istransformed to fit onto the unit circle, i.e. the Chebychev points. Thus we havethe transformationx n =cos(nπ/N) (4.5.2)

AMATH 581 ( c○J. N. Kutz) 104where n =0, 1, 2,...,N. This helps to greatly reduce the effects of the Rungephenomena.The clustered grid approximation results in a polynomial approximationshown in Fig. 30(c)-(d). There are reasons for this great improvement using aclustered grid [10]. However, we will not discuss them since they are beyondthe scope of this course. This clustered grid suggests an accurate and easy wayto represent a function which does not have periodic boundaries.Clustered Points and Chebychev DifferentiationThe clustered grid given in method 3 above is on the Chebychev points. Theresulting algorithm for constructing the polynomial approximation and differentiatingit is as follows.1. Let p be a unique polynomial of degree ≤ N with p(x n )=V n ,0≤ n ≤ Nwhere V (x) isthefunctionweareapproximating.2. Differentiate by setting w n = p ′ (x n ).The second step in the process of calculating the derivative is essentially thematrix multiplicationw = D N v (4.5.3)where D N represents the action of differentiation. By using interpolation of theLagrange form [5], the matrix elements of p(x) canbeconstructedalongwiththe (N +1)× (N +1)matrixD N .Thisresultsineachmatrixelement(D N ) ijbeing given by(D N ) 00 = 2N 2 +16(D N ) NN = − 2N 2 +16x j(4.5.4a)(4.5.4b)(D N ) jj = −2(1 − x 2 j ) j =1, 2,...,N − 1 (4.5.4c)(D N ) ij = c i(−1) i+jc j (x i − x j )i, j =0, 1,...,N (i ≠ j) (4.5.4d)where the parameter c j =2forj =0orN or c j =1otherwise.Calculating the individual matrix elements results in the matrix D N⎛⎞D N =⎜⎝2N 2 +16−(−1) i2(1−x i)−(−1) N22(−1) j1−x j. .. (−1) i+j(−1) N2x i−x j−x j(−1) N +i2(1−x 2 j ) 2(1+x j)(−1) i+jx i−x j. ..−2(−1) N +j1+x j− 2N 2 +16⎟⎠(4.5.5)

AMATH 581 ( c○J. N. Kutz) 105which is a full matrix, i.e. it is not sparse. To calculate second, third, fourth,and higher derivatives, simply raise the matrix to the appropriate power:• D 2 N - second derivative• D 3 N - third derivative• D 4 N - fourth derivative• D m N - m th derivativeBoundariesThe construction of the differentiation matrix D N does not explicitly includethe boundary conditions. Thus the general differentiation given by (4.5.3) mustbe modified to include the given boundary conditions. Consider, for example,the simple boundary conditionsv(−1) = v(1) = 0 . (4.5.6)The given differentiation matrix is then written as⎛ ⎞w⎛0top roww 1⎜ .=⎟ ⎜ first (D N ) last⎝ w N−1⎠ ⎝ columncolumnw Nbottom row⎞ ⎛⎟⎠ ⎜⎝v 0v 1.v N−1v N⎞. (4.5.7)⎟⎠In order to satisfy the boundary conditions, we must manually setv 0 = v N =0.Thus only the interior points in the differentiation matrix are relevant. Notethat for more general boundary conditions v(−1) = α and v(1) = β, wewouldsimply set v 0 = α and v N = β. The remaining (N − 1) × (N − 1) system isgiven by˜w = ˜D N ṽ (4.5.8)where ˜w =(w 1 w 2 ···w N−1 ) T , ṽ =(v 1 v 2 ···v N−1 ) T and we construct the matrix˜D N with the simple MATLAB command:tildeD=D(2:N,2:N)Note that the new matrix created is the old D N matrix with the top and bottomrows and the first and last columns removed.

AMATH 581 ( c○J. N. Kutz) 106Connecting to the FFTWe have already discussed the connection of the Chebychev polynomial with theFFT algorithm. Thus we can connect the differentiation matrix withtheFFTroutine. After transforming via (4.5.2), then for real data the discrete Fouriertransform can be used. For complex data, the regular FFT is used. Note thatfor the Chebychev polynomials∂T n (±1)∂x=0 (4.5.9)so that no-flux boundaries are already satisfied. To impose pinned boundaryconditions v(±1) = 0, then the differentiation matrix must be imposed as shownabove.4.6 Implementing the Chebychev TransformIn order to make use of the Chebychev polynomials, we must generate our givenfunction on the clustered grid given byx j =cos(jπ/N) j =0, 1, 2,...,N. (4.6.1)This clustering will give higher resolution near the boundaries than the interiorof the computational domain. The Chebychev differentiation matrixD N (4.6.2)can then be constructed. Recall that the elements of this matrix are given by(D N ) 00 = 2N 2 +16(D N ) NN = − 2N 2 +16x j(4.6.3a)(4.6.3b)(D N ) jj = −2(1 − x 2 j ) j =1, 2,...,N − 1 (4.6.3c)(D N ) ij = c i(−1) i+jc j (x i − x j )i, j =0, 1,...,N (i ≠ j) (4.6.3d)where the parameter c j =2forj =0orN or c j =1otherwise. Thusgiventhe number of discretization points N, wecanbuildthematrixD N and alsothe associated clustered grid. The following MATLAB code simply required thenumber of points N to generate both of these fundamental quantities. Recallthat it is assumed that the computational domain has been scaled to x ∈ [−1, 1].% cheb.m - compute the matrix D_Nfunction [D,x]=cheb(N)

AMATH 581 ( c○J. N. Kutz) 107if N==0, D=0; x=1; return; endx=cos(pi*(0:N)/N)’;c=[2; ones(N-1,1); 2].*(-1).^(0:N)’;X=repmat(x,1,N+1);dX=X-X’;D=(c*(1./c)’)./(dX+eye(N+1))); % off diagonalsD=D-diag(sum(D’));% diagonalsTo test the differentiation, we consider two functions for which we know theexact values for the derivative and second derivative. Consider thenx=[-1:0.01:1]u=exp(x).*sin(5*x);v=sech(x);The first and second derivative of each of these functions is given byux=exp(x).*sin(5*x) + 5*exp(x).*cos(5*x);uxx=-4*exp(x).*sin(5*x) + 10*exp(x).*cos(5*x);vx=-sech(x).*tanh(x);vxx=sech(x)-2*sech(x).^3;We can also use the Chebychev differentiation matrix to numerically calculatethe values of the first and second derivative. All that is required is the numberof discretation points N and the routine cheb.m.N=20[D,x2]=cheb(N) % x2-clustered grid, D-differentiation matrixD2=D^2; % D2 - second derivative matrixGiven the differentiation matrix D N and clustered grid, the given function andits derivatives can be constructed numerically.u2=exp(x2).*sin(5*x2);v2=sech(x2);u2x=D*u2;v2x=D*v2;u2xx=D2*u2;v2xx=D2*v2;% first derivatives% second derivativesAcomparisonbetweentheexactvaluesforthedifferentiatedfunctions and theirapproximations is shown in Fig. 31. As is expected the agreement is best forhigher values of N. The MATLAB code for generating these graphical figuresis given by

AMATH 581 ( c○J. N. Kutz) 108ydy/dxd 2 y/dx 2210-1y=exp(x)sin(5x)-2-3-1 -0.5 0 0.5 150-5-10-15-20-180-0.5 0 0.5 16040200-20-40-1 -0.5 0x0.5 110.90.80.7y=sech(x)0.6-1 -0.5 0 0.5 10.40.20-0.2-0.4-1 -0.5 0 0.5 10-0.2-0.4-0.6-0.8-1-1 -0.5 0x0.5 1Figure 31: The function y(x) =exp(x)sin5x (left) and y(x) =sechx (right)for x ∈ [−1, 1] and their first and second derivatives. The dots indicate thenumerical values while the solid line is the exact solution. For these calculations,N =20inthedifferentiationmatrixD N .figure 1; plot(x,u,’r’,x2,u2,’.’,x,v,’g’,x2,v2,’.’);figure 2; plot(x,ux,’r’,x2,u2x,’.’,x,vx,’g’,x2,v2x,’.’):figure 3; plot(x,uxx,’r’,x2,u2xx,’.’,x,vxx,’g’,x2,v2xx,’.’);Differentiation matrix in 2-DUnlike finite difference schemes which result in sparse differentiation matrices,the Chebychev differentiation matrix is full. The construction of 2D differentiationmatrices thus would seem to be a complicated matter. However, the useof the kron command is MATLAB makes this calculation trivial. In particular,the 2D Laplacian operator L given bycan be constructed withL = kron(I,D2) + kron(D2,I)Lu = ∂2 u∂x 2 + ∂2 u∂y 2 (4.6.4)

AMATH 581 ( c○J. N. Kutz) 109where D2 =D 2 and I is the identity matrix of size N × N. TheD2 ineachslottakes the x and y derivatives respectively.Solving PDEs with the Chebychev differentiation matrixTo illustrate the use of the Chebychev differentiation matrix, the two dimensionalheat equation is considered:∂u∂t = ∇2 u = ∂2 u∂x 2 + ∂2 u∂y 2 (4.6.5)on the domain x, y ∈ [−1, 1] with the Dirichlet boundary conditions u =0.The number of Chebychev points is first chosen and the differentiation matrixconstructed in one dimension.clear all; close all;N=30;[D,x]=cheb(N); D2=D^2; y=x;The Dirichlet boundary conditions are then imposed by modifying the first andlast rows of the differentiation matrix. Specifically, these rows are set to zero.D22=D2;D22(N+1,:)=zeros(1,N+1);D22(1,:)=zeros(1,N+1);The two dimensional Laplacian is then constructed from the one dimensionaldifferentiation matrix by using the kron commandI=eye(length(D22));L=kron(I,D22)+kron(D22,I);% 2D LaplacianThe initial conditions for this simulation will be a Gaussian centeredattheorigin.[X,Y]=meshgrid(x,y);U=exp(-(X.^2+Y.^2)/0.1);surfl(x,y,U), shading interp, colormap(gray), drawnowNote that the vectors x and y in the meshgrid command correspond to theChebychev points for a given N. The final step is to build a loop which willadvance and plot the solution in time. This requires the use of thereshapecommand since we are in two dimensions and an ODE solver such as ode23.

AMATH 581 ( c○J. N. Kutz) 110t=0 t=0.051 1uu0 0-1 -1x1yx1 -1t=0.1t=0.21 -1y111uu00-1 -1x1y1 -1x1 -1y1Figure 32: Evolution of an initial two dimensional Gaussian governed by theheat equation on the domain x, y ∈ [−1, 1] with Dirichlet boundary conditionsu =0. TheChebychevdifferentiationmatrixisusedtocalculatetheLaplacianwith N =30.u=reshape(U,(N+1)^2,1);for j=1:4[t,ysol]=ode23(’heatrhs2D’,[0 0.05],u,[],L);u=ysol(end,:);U=reshape(u,N+1,N+1);surfl(x,y,U), shading interp, colormap(gray), drawnowendThis code will advance the solution ∆t =0.05 in the future and plot the solution.Figure 32 depicts the two dimensional diffusive behavior given the Dirichletboundary conditions using the Chebychev differentiation matrix.4.7 Operator splitting techniquesWith either the finite difference or spectral method, the governing partial differentialequations are transformed into a system of differential equations which areadvanced in time with any of the standard time-stepping schemes. A von Neu-

AMATH 581 ( c○J. N. Kutz) 111mann analysis can often suggest the appropriateness of a scheme. For instance,we have the following:A. wave behavior: ∂u/∂t = ∂u/∂x• forward Euler: unstable for all λ• leap-frog (2,2): stable λ ≤ 1B. diffusion behavior: ∂u/∂t = ∂ 2 u/∂x 2• forward Euler: stable λ ≤ 1/2• leap-frog (2,2): unstable for all λThus the physical behavior of the governing equation dictates the kind of schemewhich must be implemented. But what if we wanted to consider the equation∂u∂t = ∂u∂x + ∂2 u∂x 2 . (4.7.1)This has elements of both diffusion and wave propagation. What time-steppingscheme is appropriate for such an equation. Certainly the Euler method seemsto work well for diffusion, but destabilizes wave propagation. In contrast, leapfrog(2,2) works well for wave behavior and destabilizes diffusion.Operator SplittingThe key idea behind operator splitting is to decouple the various physical effectsof the problem from each other. Over very small time-steps, for instance,one can imagine that diffusion would essentially act independently of the wavepropagation and vice-versa in (4.7.1). Just as in (4.7.1), the advection-diffusioncan also be thought of as decoupling. So we can consider∂ω∂t +[ψ, ω] =ν∇2 ω (4.7.2)where the bracketed terms represent the advection (wave propagation) and theright hand side represents the diffusion. Again there is a combination of wavedynamics and diffusive spreading.Over a very small time interval ∆t, itisreasonabletoconjecturethatthediffusion process is independent of the advection. Thus we could split the calculationinto the following two pieces:∆t :∆t :∂ω+[ψ, ω] =0 advectiononly (4.7.3a)∂t∂ω∂t = ν∇2 ω diffusion only . (4.7.3b)

AMATH 581 ( c○J. N. Kutz) 112This then allows us to time-step each physical effect independently over ∆t.Advantage can then be taken of an appropriate time-stepping scheme which isstable for those particular terms. For instance, in this decoupling we could solvethe advection (wave propagation) terms with a leap-frog (2,2) scheme and thediffusion terms with a forward Euler method.Additional Advantages of SplittingThere can be additional advantages to the splitting scheme. To see this, weconsider the nonlinear Schrödinger equationi ∂u∂t + 1 ∂ 2 u2 ∂x 2 + |u|2 u =0 (4.7.4)where we split the operations so thatI. ∆t : i ∂u∂t + 1 ∂ 2 u2 ∂x 2 =0(4.7.5a)II. ∆t : i ∂u∂t + |u|2 u =0.(4.7.5b)Thus the solution will be decomposed into a linear and nonlinear part. We beginby solving the linear part I. Fourier transforming yieldsi dûdt − k2û =0 (4.7.6)2which has the solution)û = û 0 exp(−i k22 t . (4.7.7)So at each step, we simply need to calculate the transform of the initial conditionû 0 ,multiplybyexp(−ik 2 t/2), and then invert.The next step is to solve the nonlinear evolution II. over a time-step ∆t.This is easily done since the nonlinear evolution admits the exact solutionu = u 0 exp(i|u 0 | 2 t) (4.7.8)where u 0 is the initical condition. This part of the calculation then requires nocomputation, i.e. we have an exact, analytic solution. The fact that we cantake advantage of this property is why the splitting algorithm is so powerful.To summarize then, the split-step method for the nonlinear Schrödingerequation yields the following algorithm.1. dispersion: u 1 =FFT −1 [ û 0 exp(−ik 2 ∆t/2) ]2. nonlinearity: u 2 = u 1 exp(i|u 1 | 2 ∆t)3. solution: u(t +∆t) =u 2Note the advantage that is taken of the analytic properties of thesolutionofeach aspect of the split operators.

AMATH 581 ( c○J. N. Kutz) 113Symmetrized SplittingAlthough we will not perform an error analysis on this scheme. It is not hardto see that the error will depend heavily on the time-step ∆t. Thusourschemefor solving∂u= Lu + N(u) (4.7.9)∂tinvolves the splittingI. ∆t :II. ∆t :∂u+ Lu =0∂t (4.7.10a)∂u+ N(u) =0.∂t (4.7.10b)To drop the error down by another O(∆t), we use what is called a Strangsplitting technique [11] which is based upon the Trotter product formula. Thisessentially involves symmetrizing the splitting algorithm sothatI.∆t2 : ∂u∂tII. ∆t :+ Lu =0(4.7.11a)∂u+ N(u) =0∂t (4.7.11b)III. ∆t2 : ∂u+ Lu =0∂t (4.7.11c)This essentially cuts the time-step in half and allows the error to drop down anorder of magnitude. It should always be used so as to keep the error in check.5 Finite Element MethodsThe numerical methods considered thus far, i.e. finite difference and spectral,are powerful tools for solving a wide variety of physical problems. However,neither method is well suited for complicated boundary configurations or complicatedboundary conditions associated with such domains. Thefiniteelementmethod is ideally suited for complex boundaries and very general boundaryconditions. This method, although perhaps not as fast as finite difference andspectral, is instrumental in solving a wide range of problems whichisbeyondthe grasp of other techniques.5.1 Finite element basisThe finite difference method approximates a solution by discretizing and solvingthe matrix problem Ax = b. Spectral methods use the FFT to expand thesolution in global sine and consine basis functions. Finite elements are anotherform of finite difference in which the approximation to the solution is made

AMATH 581 ( c○J. N. Kutz) 114by interpolating over patches of our solution region. Ultimately, to find thesolution we will once again solve a matrix problem Ax = b. Fiveessentialstepsare required in the finite element method1. Discretization of the computational domain2. Selection of the interpolating functions3. Derivation of characteristic matrices and vectors4. Assembly of characteristic matrices and vectors5. Solution of the matrix problem Ax = bAs will be seen, each step in the finite element method is mathematically significantand relatively challenging. However, there are a large number of commerciallyavailable packages that take care of implementing theaboveideas.Keep in mind that the primary reason to develop this technique isboundaryconditions: both complicated domains and general boundary conditions.Domain DiscretizationFinite element domain discretization usually involves the use of a commercialpackage. Two commonly used packages are the MATLAB PDE Toolbox andFEMLAB (which is built on the MATLAB platform). Writing your own code togenerate an unstructured computational grid is a difficult task and well beyondthe scope of this course. Thus we will use commercial packages togeneratethecomputational mesh necessary for a given problem. The key idea is to discretizethe domain with triangular elements (or another appropriate elementfunction).Figure 33 shows the discretization of a domain with complicated boundariesinside the computationally relevant region. Note that the triangular discretizationis such that it adaptively sizes the triangles so that higher resolution isautomatically in place where needed. The key features of the discretization areas follows:• The width and height of all discretization triangles should be similar.• All shapes used to span the computational domain should be approximatedby polygons.• Acommercialpackageisalmostalwaysusedtogeneratethegrid unlessyou are doing research in this area.

AMATH 581 ( c○J. N. Kutz) 115Figure 33: Finite element triangular discretization provided by the MATLABPDE Toolbox. Note the increased resolution near corners and edges.Interpolating FunctionsEach element in the finite element basis relies on a piecewise approximation.Thus the solution to the problem is approximated by a simple function in eachtriangular (or polygonal) element. As might be expected, the accuracyofthescheme depends upon the choice of the approximating function chosen.Polynomials are the most commonly used interpolating functions. The simplerthe function, the less computationally intensive. However, accuracy isusually increased with the use of higher order polynomials. Three groups ofelements are usually considered in the basic finite element scheme• simplex elements: linear polynomials are used• complex elements: higher order polynomials are used• multiplex elements: rectangles are used instead of trianglesAn example of each of the three finite elements is given in Figs. 34and35forone-dimensional and two-dimensional finite elements.

AMATH 581 ( c○J. N. Kutz) 116φ =a+bxφ =a+bx+cx 2φ exact (x)φexact(x)simplexcomplexFigure 34: Finite element discretization for simplex and complex finite elements.Essentially the finite elements are approximating polynomials.φ =a+bx+cyφ =a+bx+cy+dx +exy+fy22φ =a+bx+cy+dxysimplexcomplexmultiplexFigure 35: Finite element discretization for simplex, complex, and multiplexfinite elements in two-dimensions. The finite elements are approximating surfaces.1D SimplexTo consider the finite element implementation, we begin by considering theone-dimensional problem. Figure 36 shows the linear polynomial used to approximatethe solution between points x i and x j . The approximation of thefunction is thus( )aφ(x) =a + bx =(1 x) . (5.1.1)bThe coefficients a and b are determined by enforcing that the function goesthrough the end points. This givesφ i = a + bx iφ j = a + bx j(5.1.2a)(5.1.2b)

AMATH 581 ( c○J. N. Kutz) 117φ (x)φ iφ =a+bxφexact(x)φ jx ix jFigure 36: One dimensional approximation using finite elements.which is a 2 × 2systemfortheunknowncoefficients. Inmatrixformthiscanbe written( ) ( ) ( )φi1 xiaφ = Aa → φ = , A =, a = . (5.1.3)φ j 1 x j bSolving for a gives)( )φi= 1 (xj −x iφ j l −1 1)(φiφ j)(a = A −1 1 xj −xφ =ix j − x i −1 1(5.1.4)where l = x j − x i .Recallingthatφ =(1 x)a then givesφ = 1 ( )( )l (1 x) xj −x i φi. (5.1.5)−1 1 φ jMultiplying this expression out yields the approximate solutionφ(x) =N i (x)φ i + N j (x)φ j (5.1.6)

AMATH 581 ( c○J. N. Kutz) 118where N i (x) andN j (x) aretheLagrangepolynomialcoefficients[5](shapefunctions)N i (x) = 1 l (x j − x)N j (x) = 1 l (x − x i) .(5.1.7a)(5.1.7b)Note the following properties N i (x i )=1,N i (x j )=0andN j (x i )=0,N j (x j )=1. This completes the generic construction of the polynomial whichapproximatesthe solution in one dimension.2D simplexIn two dimensions, the construction becomes a bit more difficult. However, thesame approach is taken. In this case, we approximate the solution over a regionwith a plane (see Figure 37)φ(x) =a + bx + cy . (5.1.8)The coefficients a, b, andc are determined by enforcing that the function goesthrough the end points. This givesφ i = a + bx i + cy iφ j = a + bx j + cy jφ k = a + bx k + cy k(5.1.9a)(5.1.9b)(5.1.9c)which is now a 3 × 3systemfortheunknowncoefficients. Inmatrixformthiscan be written⎛φ = Aa → φ = ⎝ φ ⎞ ⎛iφ j⎠ , A = ⎝ 1 x ⎞ ⎛i y i1 x j y j⎠ , a = ⎝ a ⎞b ⎠ . (5.1.10)φ k1 x k y k cThe geometry of this problem is reflected in Fig. 37 where each of the points ofthe discretization triangle is illustrated.Solving for a gives⎛⎛ ⎞a = A −1 φ = 12S⎝ x ⎞jy k − x k y j x k y i − x i y k x i y j − x j y iy j − y k y k − y i y i − y j⎠x k − x j x i − x k x j − x i⎝ φ iφ j(5.1.11)where S is the area of the triangle projected onto the x − y plane (see Fig. 37.Recalling that φ = Aa givesφ(x) =N i (x, y)φ i + N j (x, y)φ j + N k (x, y)φ k (5.1.12)φ k⎠

AMATH 581 ( c○J. N. Kutz) 119φ (x,y)exact solutionφ kφ iφ japproximationφ=a+bx+cyxtriangle with( x k , y k )area S( x i , y i )( x j , y j )yFigure 37: Two dimensional triangular approximation of the solution using thefinite element method.where N i (x, y), N j (x, y), N k (x, y) aretheLagrangepolynomialcoefficients[5](shape functions)N i (x, y) = 12S [(x jy k − x k y j )+(y j − y k )x +(x k − x j )y] (5.1.13a)N j (x, y) = 12S [(x ky i − x i y k )+(y k − y i )x +(x i − x k )y] (5.1.13b)N k (x, y) = 12S [(x iy j − x j y i )+(y i − y j )x +(x j − x i )y] . (5.1.13c)Note the following end point properties N i (x i ,y i )=1,N j (x j ,y j )=N k (x k ,y k )=0andN j (x j ,y j )=1,N j (x i ,y i )=N j (x k ,y k )=0andN k (x k ,y k )=1,N k (x i ,y i )=N k (x j ,y j )=0. Thiscompletesthegenericconstructionofthepolynomial whichapproximates the solution in one dimension. This completes the generic constructionof the surface which approximates the solution in two dimensions.

AMATH 581 ( c○J. N. Kutz) 1205.2 Discretizing with finite elements and boundariesIn the previous lecture, an outline was given for the construction of the polynomialswhich approximate the solution over a finite element. Once constructed,however, they must be used to solve the physical problem of interest. The finiteelement solution method relies on solving the governing set of partial differentialequations in the weak formulation of the problem, i.e. the integral formulationof the problem. This is because we will be using linear interpolation pieceswhose derivatives don’t necessarily match accross elements. In the integral formulation,this does not pose a problem.We begin by considering the elliptic partial differential equation∂∂x(p(x, y) ∂u∂x)+ ∂ ∂ywhere over part of the boundaryand(q(x, y) ∂u∂y)+ r(x, y)u = f(x, y) (5.2.1)u(x, y) =g(x, y) on S 1 (5.2.2)p(x, y) ∂u∂x cos θ 1 + q(x, y) ∂u∂y cos θ 2 + g 1 (x, y)u = g 2 (x, y) on S 2 . (5.2.3)The boundaries and domain are illustrated in Fig. 38. Note that the normalderivative determines the angles θ 1 and θ 2 . A domain such as this would bevery difficult to handle with finite difference techniques. And further, becauseof the boundary conditions, spectral methods cannot be implemented. Only thefinite element method renders the problem tractable.The Variational PrincipleTo formulate the problem correctly for the finite element method, the governingequations and its associated boundary conditions are recast inanintegralform.This recasting of the problem involves a variational principle. Although we willnot discuss the calculus of variations here [12], the highlights of this methodwill be presented.The variational method expresses the governing partial differential as anintegral which is to be minimized. In particular, the functional to be minimizedis given by∫∫∫ (I(φ) = F φ, ∂φV ∂x , ∂φ∂y , ∂φ ) ∫∫ (dV + g φ, ∂φ∂zS ∂x , ∂φ∂y , ∂φ )dS , (5.2.4)∂zwhere a three dimensional problem is being considered. The derivation of thefunctions F , which captures the governing equation, and g, which captures

AMATH 581 ( c○J. N. Kutz) 121ynormalS 1θ 2θ1S 2Figure 38: Computational domain of interest which has the two domainregionsS 1 and S 2 .xthe boundary conditions, follow the Euler-Lagrange equations for minimizingvariations:δFδφ = ∂ ( ) ∂F+ ∂ ( ) ∂F+ ∂ ( ) ∂F− ∂F =0. (5.2.5)∂x ∂(φ x ) ∂y ∂(φ y ) ∂z ∂(φ z ) ∂φThis is essentially a generalization of the concept of the zero derivative whichminimizes a function. Here we are minimizing a functional.For our governing elliptic problem (5.2.1) with boundary conditions givenby those of S 2 (5.2.3), we find the functional I(u) tobegivenbyI(u) = 1 ∫∫ [ ( ) 2 ( ) 2 ∂u∂udxdy p(x, y) +q(x, y) −r(x, y)u 2 +2f(x, y)u]2 D∂x∂y∫+ dS[−g 2 (x, y)u + 1 ]S 22 g 1(x, y)u 2 . (5.2.6)

AMATH 581 ( c○J. N. Kutz) 122Thus the interior domain over which we integrate D has the integrand[I D = 1 ( ) 2 ( ) 2 ∂u∂up(x, y) +q(x, y) −r(x, y)u 2 +2f(x, y)u]. (5.2.7)2 ∂x∂yThis integrand was derived via variational calculus as the minimization over thefunctional space of interest. To confirm this, we apply the variational derivativeas given by (5.2.5) and findδI Dδu = ∂∂x= ∂∂x= ∂∂x( ) ∂ID+ ∂ ( ) ∂ID− ∂I D∂(u x ) ∂y ∂(u y ) ∂u =0( )1 ∂up(x, y) · 2 + ∂ ( 1 ∂uq(x, y)·22 ∂x ∂y 2 ∂y(p(x, y) ∂u )+ ∂ (q(x, y) ∂u∂x ∂y ∂y)−(− 1 2 r(x, y)·2u+f(x, y) ))+r(x, y)u−f(x, y) =0. (5.2.8)Upon rearranging, this gives back the governing equation (5.2.1)(∂p(x, y) ∂u )+ ∂ (q(x, y) ∂u )+ r(x, y)u = f(x, y) . (5.2.9)∂x ∂x ∂y ∂yAsimilarprocedurecanbefollowedontheboundarytermstoderive the appropriateboundary conditions (5.2.2) or (5.2.3). Once the integral formulation hasbeen achieved, the finite element method can be implemented with the followingkey ideas:1. The solution is expressed in the weak (integral) form since the linear (simplex)interpolating functions give that the second derivates (e.g. ∂ 2 x and∂ 2 y)arezero. Thustheellipticoperator(5.2.1)wouldhavenocontributionfrom a finite element of the form φ = a 1 + a 2 x + a 3 y. However, inthe integral formulation, the second derivative terms are proportional to(∂u/∂x) 2 +(∂u/∂y) 2 which gives φ = a 2 2 + a 2 3.2. A solution is sought which takes the formu(x, y) =m∑γ i φ i (x, y) (5.2.10)i=1where φ i (x, y) arethelinearlyindependentpiecewise-linearpolynomialsand γ 1 ,γ 2 ,...,γ m are constants.• γ n+1 ,γ n+2 ,...,γ m ensure that the boundary conditions are satisfied,i.e. those elements which touch the boundary must meet certainrestrictions.• γ 1 ,γ 2 ,...,γ n ensure that the integrand I(u) intheinteriorofthecomputationaldomain is minimized, i.e. ∂I/∂γ i =0fori =1, 2, 3,...,n.

AMATH 581 ( c○J. N. Kutz) 123Solution MethodWe begin with the governing equation in integral form (5.2.6)andassumeanexpansion of the form (5.2.10). This gives( m)∑I(u) = I γ i φ i (x, y)= 1 2i=1∫∫D⎡dxdy ⎣p(x, y)( m∑i=1) 2∂φ iγ i +q(x, y)∂x( m∑i=1γ i∂φ i∂y( m 2 (∑∑ m−r(x, y) γ i φ i (x, y))+2f(x, y) γ i φ i (x, y)) ⎤ ⎦ (5.2.11)∫+ dSS 2⎡i=1⎣−g 2 (x, y)i=1) 2(m∑γ i φ i (x, y)+ 1 m) ⎤ 22 g ∑1(x, y) γ i φ i (x, y) ⎦ .i=1This completes the expansion portion.The functional I(u)mustnowbedifferentiatedwithrespecttoalltheinteriorpoints and minimized. This involves differentiating the above with respect toγ i where i =1, 2, 3,...,n.Thisresultsintherathercomplicatedexpression∂I∂γ j=∫∫Ddxdy−r(x, y)[m∑ ∂φ i ∂φ jp(x, y) γ i∂xi=1]m∑γ i φ i φ j +f(x, y)φ ji=1∫ [+ dS −g 2 (x, y)φ j + g 1 (x, y)S 2i=1∂x +q(x, y) m ∑i=1γ i∂φ i∂y]m∑γ i φ i φ j =0.i=1∂φ j∂y(5.2.12)Term by term, this results in an expression for the γ i given bym∑{∫ ∫ [dxdy p ∂φ i ∂φ ji=1 D ∂x ∂x +q ∂φ ]}i ∂φ j∂y ∂y −rφ iφ j + dSg 1 φ i φ j γ i∫S 2∫∫∫+ dxdyfφ j − dSg 2 (x, y)φ j =0. (5.2.13)DS 2But this expression is simply a matrix solve Ax = b for the unknowns γ i . Inparticular, we have⎛ ⎞ ⎛ ⎞γ 1β 1γ 2x = ⎜ ⎟⎝ . ⎠b = β 2⎜ ⎟⎝ . ⎠ A =(α ij) (5.2.14)γ nβ n

AMATH 581 ( c○J. N. Kutz) 124where∫∫∫β i =− dxdyfφ i +D∫∫α ij = dxdyDm∑dSg 2 φ i − α ik γ k(5.2.15a)S 2 k=n+1[p ∂φ i ∂φ j∂x ∂x +q ∂φ ] ∫i ∂φ j∂y ∂y −rφ iφ j + dSg 1 φ i φ j . (5.2.15b)S 2Recall that each element φ i is given by the simplex approximationφ i =3∑i=1N (i)j(x, y)φ (i)j=3∑i=1(a (i)j)+ b (i)jx + c(i) j y φ (i)j. (5.2.16)This concludes the construction of the approximate solutioninthefiniteelementbasis. From an algorithmic point of view, the following procedure would becarried out to generate the solution.1. Discretize the computational domain into triangles. T 1 ,T 2 ,...,T k are theinterior triangles and T k+1 ,T k+2 ,...,T m are triangles that have at leastone edge which touches the boundary.2. For l = k +1,k +2,...,m,determinethevaluesoftheverticesonthetriangles which touch the boundary.3. Generate the shape functionsN (i)j= a (i)jwhere i =1, 2, 3,...,m and j =1, 2, 3,...,m.+ b (i)j x + c(i) j y (5.2.17)4. Compute the integrals for matrix elements α ij and vector elements β j inthe interior and boundary.5. Construct the matrix A and vector b.6. Solve Ax = b.7. Plot the solution u(x, y) = ∑ mi=1 γ iφ i (x, y).For a wide variety of problems, the above procedures can simply be automated.That is exactly what commercial packages do. Thus once the coefficients andboundary conditions associated with (5.2.1), (5.2.2), and (5.2.3) are known, thesolution procedure is straightforward.

AMATH 581 ( c○J. N. Kutz) 125AApplication ProblemsThe purpose of this class: to be able to solve realistic and difficult problemswhich arise in a variety of physical settings. This appendix outlines a fewphysical problems which give rise to phenomena of great general interest. Thetechniques developed in the lectures on finite difference, spectral, and finiteelement methods provide all the necessary tools for solving these problems.Thus each problem isn’t simply a mathematical problem, but rather a way todescribe phenomena which may be observed in various physically interestingsystems.A.1 Advection-diffusion and Atmospheric DynamicsThe shallow-water wave equations are of fundamental interest in several contexts.In one sense, the ocean can be thought of as a shallow water descriptionover the surface of the earth. Thus the circulation and movement of currentscan be studied. Second, the atmosphere can be thought of as a relatively thinlayer of fluid (gas) above the surface of the earth. Again the circulation andatmospheric dynamics are of general interest. The shallow-water approximationrelies on a separation of scale: the height of the fluid (or gas) mustbemuchlessthan the characteristic horizontal scales. The physical setting for the shallowwatermodeling is illustrated in Fig. 39. In this figure, the characteristic heightis given by the parameter D while the characteristic horizontal fluctuations aregiven by the parameter L. For the shallow-water approximation to hold, wemust haveδ = D L ≪ 1 .(A.1.1)This gives the necessary separation of scales for reducing the Navier-Stokesequations to the shallow-water description.The motion of the layer of fluid is described by its velocity field⎛ ⎞v =⎝ u vw⎠(A.1.2)where u, v, and w are the velocities in the x, y, and z directions respectively. Analternative way to describe the motion of the fluid is through the quantity knownas the vorticity. Roughly speaking the vorticity is a vector which measures thetwisting of the fluid, i.e. Ω =(ω x ω y ω z ) T . Since with shallow-water we areprimarily interested in the vorticity or fluid rotation in the x−y plane, we definethe vorticity of interest to beω z = ω = ∂v∂x − ∂u∂y .(A.1.3)This quantity will characterize the evolution of the fluid in the shallow-waterapproximation.

AMATH 581 ( c○J. N. Kutz) 126z (w)LΩy (v)ρ =constanth(x,y,z,t)DFigure 39: Physical setting of a shallow atmosphere or shallow water equationswith constant density ρ and scale separation D/L ≪ 1. The vorticity ismeasured by the parameter Ω.x (u)Conservation of MassThe equations of motion are a consequence of a few basic physical principles, oneof those being conservation of mass. The implications of conservation of mass:the time rate of change of mass in a volume must equal the net inflow/outflowthrough the boundaries of the volume. Consider a volume from Fig. 39 which isbounded between x ∈ [x 1 ,x 2 ]andy ∈ [y 1 ,y 2 ]. The mass in this volume is givenbymass =∫ x2∫ y2x 1y 1ρ(x, y)h(x, y, t)dxdy (A.1.4)where ρ(x, y) isthefluiddensityandh(x, y, t) isthesurfaceheight. Wewillassume the density ρ is constant in what follows. The conservation of mass inintegral form is then expressed as∂∂t∫ x2∫ y2x 1 y∫1y2++y∫1x2x 1h(x, y, t)dxdy[u(x 2 ,y,t)h(x 2 ,y,t) − u(x 1 ,y,t)h(x 1 ,y,t)] dy[v(x, y 2 ,t)h(x, y 2 ,t)−v(x, y 1 ,t)h(x, y 1 ,t)] dx =0, (A.1.5)where the density has been divided out. Here the first term measures the rateof change of mass while the second and third terms measure the flux of massacross the x boundaries and y boundaries respectively. Note that from the

AMATH 581 ( c○J. N. Kutz) 127fundamental theorem of calculus, we can rewrite the second and third terms:∫ y2y 1[u(x 2 ,y,t)h(x 2 ,y,t)−u(x 1 ,y,t)h(x 1 ,y,t)] dy =∫ x2x 1[v(x, y 2 ,t)h(x, y 2 ,t)−v(x, y 1 ,t)h(x, y 1 ,t)] dx=∫ x2∫ y2x 1 y∫1x2∫ y2x 1y 1∂(uh)dxdy (A.1.6a)∂x∂(vh)dxdy. (A.1.6b)∂yReplacing these new expressions in (A.1.5) shows all terms to haveadoubleintegral over the volume. The integrand must then be identically zero for conservationof mass. This results in the expression∂h∂t + ∂∂x (hu)+ ∂ (hv) =0.∂y (A.1.7)Thus a fundamental relationship is established from a simple firstprinciplesargument. The second and third equations of motion for the shallow-waterapproximation result from the conservation of momentum in the x and y directions.These equations may also be derived directly from the Navier-Stokesequations or conservation laws.Shallow-Water EquationsThe conservation of mass and momentum generates the following three governingequations for the shallow-water description∂h∂t + ∂∂x (hu)+ ∂ ∂y∂∂t (hu)+ ∂∂x∂∂t (hv)+ ∂ ∂y(hv) =0(A.1.8a)(hu 2 + 1 )2 gh2 + ∂ (huv) =fhv (A.1.8b)∂y(hv 2 + 1 )2 gh2 + ∂ (huv) =−fhu.∂x (A.1.8c)Various approximations are now used to reduce the governing equations to amore manageable form. The first is to assume that at leading order the fluidheight h(x, y, t) isconstant. Theconservationofmassequation(A.1.8a)thenreduces to∂u∂x + ∂v∂y =0,(A.1.9)which is referred to as the imcompressible flow condition. Thus under thisassumption, the fluid cannot be compressed.Under the assumption of a constant height h(x, y, t), the remaining twoequations reduce to∂u ∂u+2u∂t ∂x + ∂ (uv) =fv∂y (A.1.10a)∂v ∂v+2v∂t ∂y + ∂ (uv) =−fu.∂x (A.1.10b)

AMATH 581 ( c○J. N. Kutz) 128To simplify further, take the y derivative of the first equation and the x derivativeof the second equation. The new equations are∂ 2 u∂t∂y +2∂u ∂u∂y ∂x +2u ∂2 u∂x∂y + ∂2 ∂v(uv) =f∂y2 ∂y∂ 2 v∂t∂x +2∂v ∂v∂x ∂y +2v ∂2 v∂x∂y + ∂2∂u(uv) =−f∂x2 ∂x .(A.1.11a)(A.1.11b)Subtracting the first equation from the second gives the following reductions(∂ ∂v∂t ∂x − ∂u )− 2 ∂u ∂u∂y ∂y ∂x − 2u ∂2 u∂x∂y − ∂2∂y 2 (uv)+2 ∂v(∂v∂x ∂y +2v ∂2 v∂x∂y + ∂2∂u∂x 2 (uv) =−f ∂x + ∂v )∂y(∂ ∂v∂t∂x − ∂u∂y+u(∂ ∂v∂t)+2 ∂u∂x( ∂ 2 v∂x 2 − ∂2 v∂y 2 − 2 ∂2 u∂x − ∂u∂y+v ∂∂x)+ u ∂( ∂u∂x + ∂v∂y( ∂v∂x − ∂u )+2 ∂v ( ∂v∂y ∂y ∂x − ∂u )(A.1.12)∂y) ( ∂ 2 ) (u+ v∂x∂y ∂x 2 − ∂2 u∂y 2 +2 ∂2 v ∂u= −f∂x∂y ∂x + ∂v )∂y( ∂v∂x ∂x − ∂u )+ v ∂ ( ∂v∂y ∂y ∂x − ∂u )− u ∂ ( ∂u∂y ∂y ∂x + ∂v )∂y) ( ∂v+2∂x − ∂u )( ∂u∂y ∂x + ∂v ) ( ∂u= −f∂y ∂x + ∂v ).∂yThe final equation is reduced greatly by recalling the definition of the vorticity(A.1.3) and using the imcompressibility condition (A.1.9). The governingequations then reduce to∂ω∂t + u ∂ω∂x + v ∂ω∂y =0.(A.1.13)This gives the governing evolution of a shallow-water fluid in theabsenceofdiffusion.The StreamfunctionIt is typical in many fluid dynamics problems to work with the quantity knownas the streamfunction. The streamfunction ψ(x, y, t) isdefinedasfollows:u = − ∂ψ∂yv = ∂ψ∂x .(A.1.14)Thus the streamfunction is specified up to an arbitrary constant. Note that thestreamfunction automatically satisfies the imcompressibility condition since∂u∂x + ∂v∂y = − ∂2 ψ∂x∂y + ∂2 ψ∂x∂y =0.(A.1.15)

AMATH 581 ( c○J. N. Kutz) 129In terms of the vorticity, the streamfunction is related as follows:ω = ∂v∂x − ∂u∂y = ∇2 ψ.(A.1.16)This gives a second equation of motion which must be considered in solving theshallow-water equations.Advection-DiffusionThe advection of a fluid is governed by the evolution (A.1.13). Inthepresenceof frictional forces, modification of this governing equation occurs. Specifically,the motion in the shallow-water limit is given by∂ω∂t +[ψ, ω] =ν∇2 ω∇ 2 ψ = ω(A.1.17a)(A.1.17b)where[ψ, ω] = ∂ψ ∂ω∂x ∂y − ∂ψ ∂ω∂y ∂x(A.1.18)and ∇ 2 = ∂x 2 + ∂2 y is the two dimensional Laplacian. The diffusion componentwhich is proportional to ν measures the frictional forces present in the fluidmotion.The advection-diffusion equations have the characteristic behavior of thethree partial differential equations classifications: parabolic, elliptic, and hyperbolic:parabolic:∂ω∂t = ν∇2 ω(A.1.19a)elliptic: ∇ 2 ψ = ω (A.1.19b)hyperbolic:∂ω+[ψ, ω] =0.∂t (A.1.19c)Two things need to be solved for as a function of time:ψ(x, y, t) streamfunction (A.1.20a)ω(x, y, t) vorticity. (A.1.20b)We are given the initial vorticity ω 0 (x, y) andperiodicboundaryconditions.The solution procedure is as follows:1. Elliptic Solve: Solve the elliptic problem ∇ 2 ψ = ω 0 to find the streamfunctionat time zero ψ(x, y, t =0)=ψ 0 .

AMATH 581 ( c○J. N. Kutz) 1302. Time-Stepping: Given now ω 0 and ψ 0 , solve the advection-diffusionproblem by time-stepping with a given method. The Euler method isillustrated belowω(x, y, t +∆t) =ω(x, y, t)+∆t ( ν∇ 2 ω(x, y, t) − [ψ(x, y, t),ω(x, y, t)] )This advances the solution ∆t into the future.3. Loop: With the updated value of ω(x, y, ∆t), we can repeat the processby again solving for ψ(x, y, ∆t) andupdatingthevorticityonceagain.This gives the basic algorithmic structure which must be followed in order togenerate the solution for the vorticity and streamfunction as a function of time.It only remains to discretize the problem and solve.A.2 Introduction to Reaction-Diffusion SystemsTo begin a discussion of the need for generic reaction-diffusion equations, weconsider a set of simplified models relating to predator-prey population dynamics.These models consider the interaction of two species: predators and theirprey. It should be obvious that such species will have significant impact on oneanother. In particular, if there is an abundance of prey, then thepredatorpopulationwill grow due to the surplus of food. Alternatively, if thepreypopulationis low, then the predators may die off due to starvation.To model the interaction between these species, we begin by considering thepredators and prey in the absence of any interaction. Thus the preypopulation(denoted by x(t)) is governed bydxdt = ax(A.2.1)where a>0isanetgrowthconstant. Thesolutiontothissimpledifferentialequation is x(t) =x(0) exp(at) sothatthepopulationgrowswithoutbound.We have assumed here that the food supply is essentially unlimited for the preyso that the unlimited growth makes sense since there is nothing to kill off thepopulation.Likewise, the predators can be modeled in the absence of their prey.Inthiscase, the population (denoted by y(t)) is governed bydydt = −cy(A.2.2)where c>0isanetdecayconstant. Thereasonforthedecayisthatthepopulation starves off since there is no food (prey) to eat.We now try to model the interaction. Essentially, the interaction must accountfor the fact the the predators eat the prey. Such an interaction term can

AMATH 581 ( c○J. N. Kutz) 131result in the following system:dx= ax − αxydt (A.2.3a)dy= −cx + αxy ,dt (A.2.3b)where α>0istheinteractionconstant. Notethatα acts as a decay to theprey population since the predators will eat them, and as a growth term to thepredators since they now have a food supply. These nonlinear and autonomousequations are known as the Lotka–Volterra equations. Therearetwofundamentallimitations of this model: the interaction is only heuristic in nature and thereis no spatial dependence. Thus the validity of this simple modeling is certainlyquestionable.Spatial DependenceOne way to model the dispersion of a species in a given domain is byassumingthe dispersion is governed by a diffusion process. If in addition we assume thatthe prey population can saturate at a given level, then the governing populationequations are( )∂x∂t = a x − x2− αxy + D 1 ∇ 2 x(A.2.4a)k∂y∂t = −cx + αxy + D 2∇ 2 y,(A.2.4b)which are known as the modified Lotka-Volterra equations. It includes thespecies interaction with saturation, i.e. the reaction terms, and spatial spreadingthrough diffusion, i.e. the diffusion term. Thus it is a simple reaction-diffusionequation.Along with the governing equations, boundary conditions must be specified.A variety of conditions may be imposed, these include periodic boundaries,clamped boundaries such that x and y are known at the boundary, fluxboundaries in which ∂x/∂n and ∂y/∂n are known at the boundaries, or somecombination of flux and clamped. The boundaries are significant in determiningthe ultimate behavior in the system.Spiral WavesOne of the many phenomena which can be observed in reaction-diffusion systemsis spiral waves. An excellent system for studying this phenomena is theFitzhugh-Nagumo model which provides a heuristic description of an excitablenerve potential:∂u∂t = u(a − u)(1 − u) − v + D∇2 u(A.2.5a)

AMATH 581 ( c○J. N. Kutz) 132∂v∂t= bu − γv ,(A.2.5b)where a, D, b, and γ are tunable parameters.Abasicunderstandingofthespiralwavephenomenacanbeachieved byconsidering this problem in the absence of the diffusion. Thus thereactionterms alone give∂u= u(a − u)(1 − u) − v∂t (A.2.6a)∂v= bu − γv .∂t (A.2.6b)This reduces to a system of differential equations for which the fixed points canbe considered. Fixed points occur when ∂u/∂t = ∂v/∂t =0. Thethreefixedpoints for this system are given by(u, v) =(0, 0)(u, v) =(u ± , (a − u ± )(1 − u ± )u ± )(A.2.7a)(A.2.7b)where u ± =[(a +1)± ((a +1) 2 − 4(a − b/γ)) 1/2 ]/2.The stability of these three fixed points can be found by linearization [1].In particular, consider the behavior near the steady-state solution u = v =0.Thus letu =0+ũv =0+ṽ(A.2.8a)(A.2.8b)where ũ, ṽ ≪ 1. Plugging in and discarding higher order terms gives the linearizedequationsThis can be written as the linear system∂ũ∂t = aũ − ṽ∂ṽ= bũ − γṽ.∂t (A.2.9b)(dx adt = b−1−γ)x .(A.2.9a)(A.2.10)Assuming a solution of the form x = v exp(λt) resultsintheeigenvalueproblem( )a − λ −1v =0(A.2.11)b −γ − λwhich has the eigenvaluesλ ± = 1 [(a − γ) ± √ ](a − γ)22 +4(b − aγ) . (A.2.12)

AMATH 581 ( c○J. N. Kutz) 133ustableu+unstablexu-stableFigure 40: Front solution which connects the two stable branch of solutions u ±through the unstable solution u =0.Figure 41: Spatial-temporal evolution of the solution of the Fitzhugh-Nagumomodel for the parameter D =2× 10 −6 , a =0.25, b =10 −3 ,andγ =3× 10 −3 .The dark regions are where u = u + whereas the white regions are where u = u − .In the case where b − aγ > 0, the eigenvalues are purely real with one positiveand one negative eigenvalue, i.e. it is a saddle node. Thus the steady-statesolution in this case is unstable.Further, if the condition b−aγ > 0holds,thenthetworemaningfixedpointsoccur for u − < 0andu + > 0. The stability of these points may also be found.For the parameter restrictions considered here, these two remaining points arefound to be stable upon linearization.The question which can then naturally arise: if the two stable solutionsu ± are connected, how will the front between the two stable solutions evolve?The scenario is depicted in one dimension in Fig. 40 where the two stable u ±branches are connected through the unstable u =0solution. Figure41depictsthe spatial-temporal evolution of the solution of the Fitzhugh-Nagumo modelfor the parameter D =2×10 −6 , a =0.25, b =10 −3 ,andγ =3×10 −3 .Thedarkregions are where u = u + whereas the white regions are where u = u − . Note

AMATH 581 ( c○J. N. Kutz) 134Figure 42: Experimental observations in which target patterns and spiral wavesare both exhibited. These behaviors are generated in the Belousov-Zhabotinskiireaction by pacemaker nuclei (a) and (b), and by the slime mold Dictyostelium(c) which emit periodic signals of the chemical cyclic AMP, which is a chemoattractantfor the cells.the formation of spiral waves. The boundary conditions used here are no flux,i.e. ∂u/∂x =0or∂u/∂y =0acrosstheleft/rightandtop/bottomboundariesrespectively.Not just an interesting mathematical phenomena, spiral waves are also exhibitedin nature. Figure 42 illustrates a series of experimental observationsin which target patterns and spiral waves are both exhibited. These behaviorsare generated in the Belousov-Zhabotinskii reaction by pacemaker nuclei andby the slime mold Dictyostelium which emit periodic signals of the chemicalcyclic AMP, which is a chemoattractant for the cells. Spiral waves are seen tobe naturally generated in these systems and are of natural interest for analyticstudy.

AMATH 581 ( c○J. N. Kutz) 135The λ − ω Reaction-Diffusion SystemThe general reaction-diffusion system can be written in the vector form∂u∂t = f(u)+D∇2 u(A.2.13)where the diffusion is proportional to the parameter D and the reaction termsare given by f(u). The specific case we consider is the λ − ω system which takesthe form∂∂t( uv) ( )( λ(A) −ω(A) u=ω(A) λ(A) v)+ D∇ 2 ( uv)(A.2.14)where A = u 2 + v 2 .Thusthenonlinearityiscubicinnature.Thisallowsforthepossibility of supporting three steady-state solutions as in the Fitzhugh-Nagumomodel which led to spiral wave behavior. The spirals to be investigated in thiscase can have one, two, three or more arms. An example of a spiral wave initialcondition is given byu 0 (x, y) =tanh|r| cos(mθ −|r|)v 0 (x, y) =tanh|r| sin(mθ −|r|)(A.2.15a)(A.2.15b)where |r| = √ x 2 + y 2 and θ = x+iy. Theparameterm determines the numberof arms on the spiral. The stability of the spiral evolution under perturbationand in different parameter regimes is essential in understanding the underlyingdynamics of the system. A wide variety of boundary conditions can alsobe applied. Namely, periodic, clamped, no-flux, or mixed. In each case, theboundaries have significant impact on the resulting evolution as the boundaryeffects creep into the middle of the computational domain.A.3 Steady State Flow Over an AirfoilThe derivation for the steady-state flow over an airfoil is similar to that of thevorticity and streamfunction dynamics in shallow-water. The physical settingof interest now is geared towards the behavior of fluids around airfoils. Inthiscase, a separation of scales is achieved by assuming the airfoil (wing) is muchlonger in length than its thickness or width. The scale separation once againallows for a two-dimensional study of the problem. Additionally, since we areconsidering the steady-state flow, the time-dependence of the problem will beeliminated. The resulting problem will be of the elliptic type. This will give thesteady-state streamfunction and flow over the wing.The physical setting for the airfoil modeling is illustrated inFig.43. Inthis figure, the characteristic wing width is given by the parameter D whilethe characteristic wing length is given by the parameter L. For a quasi-twodimensionaldescription to hold, we must haveδ = D L ≪ 1 .(A.3.1)

AMATH 581 ( c○J. N. Kutz) 136z (w)wingy (v)ρω xDω z=constant r(x,y,z,t)ω yLh(x,y,z,t)x (u)Figure 43: Physical setting of an airfoil in a constant density fluid ρ with scaleseparation D/L ≪ 1.This gives the necessary separation of scales for reducing the Navier-Stokesequations to the shallow-water description. In particular, theshallow-waterreductionwhich led to the vorticity-streamfunction equations can be pursued here.The governing equations which result from mass conservation andmomentumconservation in the x and z planes give∂r∂t + ∂∂x (ru)+ ∂ (rw) =0∂z (A.3.2a)∂∂t (ru)+ ∂ (ru 2 + 1 )∂x 2 gr2 + ∂ (ruw) =0 (A.3.2b)∂z∂∂t (rw)+ ∂ ∂z(rw 2 + 1 )2 gr2 + ∂ (ruw) =0∂x (A.3.2c)where we have ignored the coriolis parameter.At leading order, we assume r(x, y, t) ≈constant so that the conservation ofmass equation reduces to∂u∂x + ∂w∂z =0,(A.3.3)which is again referred to as the imcompressible flow condition. Under the assumptionof a constant width r(x, z, t), the remaining two momentum equationsreduce further. The final reduction comes from taking the z derivative of thefirst equation and the x derivative of the second equation and performing thealgebra associated with (A.1.10) through (A.1.13) which gives∂ω∂t +[ψ, ω] =ν∇2 ω∇ 2 ψ = ω(A.3.4a)(A.3.4b)

AMATH 581 ( c○J. N. Kutz) 137Figure 44: Fluid flow past a cylinder for increasing Reynolds number. Thefour pictures correspond to R =9.6, 41.0, 140, and 10000. Note the transitionfrom laminar (steady-state) flow to a recirculating steady-state flow, to Kármanvortex shedding and finally to fully turbulent flow.where[ψ, ω] = ∂ψ ∂ω∂x ∂z − ∂ψ ∂ω∂z ∂x(A.3.5)and ∇ 2 = ∂x 2 + ∂z 2 is the two dimensional Laplacian. The diffusion componentwhich is proportional to ν measures the frictional forces present in the fluidmotion. Note that we have once again introduced the streamfunction ψ(x, z, t)which is defined asu = − ∂ψ w = ∂ψ∂z ∂x .(A.3.6)Thus the streamfunction is specified up to an arbitrary constant. Note that thestreamfunction automatically satisfies the imcompressibility condition since∂u∂x + ∂w∂z = − ∂2 ψ∂x∂z + ∂2 ψ∂x∂z =0.In terms of the vorticity, the streamfunction is related as follows:ω = ω y = ∂w∂x − ∂u∂z = ∇2 ψ.(A.3.7)(A.3.8)This gives a second equation of motion (A.3.4a) which must be considered insolving the airfoil equations.

AMATH 581 ( c○J. N. Kutz) 138Figure 45: Turbulent flow over a sphere for Reynolds number R =15000.Steady-State FlowThe behavior of the fluid as it propagates over the airfoil can change drasticallydepending upon the density and velocity of the fluid. The typical measure of thecombined fluid density and velocity is given by a quantity known as the Reynoldsnumber R. Essentially,asthetheReynoldsnumberincreases,thenthefluidiseither moving at higher velocities or has a lower density. The experimentalfluidflow around a cylinder, which can be thought of as an airfoil, is depictedinFig. 44. The four pictures correspond to R =9.6, 41.0, 140, and 10000. Notethe transition from laminar (steady-state) flow to a recirculating steady-stateflow, to Kárman vortex shedding and finally to fully turbulent flow.Thesametype of phenomena occurs also in three dimensions. A fully developed turbulentflow over a sphere is illustrated in Fig. 45 for R =15000.The general turbulent behavior is difficult to capture computationally. However,the steady-state behavior can be well understood by considering the equationsof motion in the steady-state limit. In particular, at large times in thesteady-state flow the diffusion term in the advection-diffusion equation (A.3.4a)causes the vorticity to diffuse to zero, i.e. as t →∞then ω → 0. The streamfunctionequation (A.3.4b) then becomes Poisson’s equation∇ 2 ψ =0.(A.3.9)Thus an elliptic solve is the only thing required to find the steady-state behavior.Boundary ConditionsTo capture the steady-state flow which is governed by the elliptic problem, weneed to develop boundary conditions for both the computational domain and air-

AMATH 581 ( c○J. N. Kutz) 139z4Δψ=01 523xFigure 46: Computational domain for calculating the steady-state flow over anairfoil. Note the five boundaries which must be specified to reach a solution.foil. Figure 46 illustrates the computational domain which may be implementedto capture the steady-state flow over an airfoil. Five boundary conditions arespecified to determine the steady-state flow over the airfoil. Thesefiveboundaryconditions on the domain x ∈ [−L, L] andy ∈ [−L, L] areasfollows:1. x = −L: The incoming flux flow it imposed at this location. Assumingthe flow enters so that u(x = −L) =u 0 and w(x = −L) =0givesu = u 0 = − ∂ψ∂z→∂ψ∂z = −u 0(A.3.10)2. x = L: Theoutgoingfluxflowshouldbeindenticaltothatimposedfor1.Thus (A.3.10) should apply.3. z = −L: Assumethatfaraboveorbelowtheairfoil,thefluidflowremainsparallel to the x direction. Thus (A.3.10) still applies. However, we alsohave thatw =0= ∂ψ∂x → ∂ψ∂x =0(A.3.11)4. z = L: The boundary conditions at the top of the domain will mirrorthose at the bottom.5. airfoil: the steady-state flow cannot peneratrate into the airfoil. Thusaboundary condition is required on the wing itself. Physically, a no slip

AMATH 581 ( c○J. N. Kutz) 140boundary condition is applied. Thus the fluid is stationary on thedomainof the wing:u = w =0 → ∂ψ∂x = ∂ψ∂z =0(A.3.12)Although the rectangular computational domain is easily handled with finitedifference techniques, the airfoil itself presents a significant challenge. In particular,the shape is not regular and the finite difference technique is only equippedto handle rectangular type domains. Thus an alternative method should be used.Specifically, finite elements can be considered since it allows for an unstructureddomain and can handle any complicated wing shape. Realistically, any attemptwith finite difference or spectral techniques on a complicated domainismoretrouble than its worth.References[1] W. E. Boyce and R. C. DiPrima, Elementary Differential Equations andBoundary Value Problems, 7thEd.(Wiley,2001).[2] See, for instance, R. Finney, F. Giordano, G. Thomas, and M. Weir, Calculusand Analytic Geometry, 10thEd.(PrenticeHall,2000).[3] C. W. Gear, Numerical Initial Value Problems in Ordinary DifferentialEquations, (PrenticeHall,1971).[4] J. D. Lambert, Computational Methods in Ordinary Differential Equations,(Wiley, 1973)[5] R. L. Burden and J. D. Faires, Numerical Analysis, (Brooks/Cole,1997).[6] A. Greenbaum, Iterative Methods for Solving Linear Systems, (SIAM,1997).[7] W. H. Press, S. A. Teukolsky, W. T. Vetterling, B. P. Flannery, NumericalRecipes, 2ndEd.(Cambridge,1992).[8] R. Courant, K. O. Friedrichs, and H. Lewy, “Über die partiellen differenzengleichungender mathematischen physik,” Mathematische Annalen 100,32-74 (1928).[9] J. C. Strikwerda, Finite Difference Schemes and Partial Differential Equations,(Chapman&Hall,1989).[10] N. L. Trefethen, Spectral methods in MATLAB, (SIAM,2000).[11] G. Strang, Introduction to Applied Mathematics, (Wellesley-CambridgePress, 1986).[12] I. M. Gelfand and S. V. Fomin, Calculus of Variations, (Prentice-Hall,1963).