NUMERICAL DYNAMIC PROGRAMMING

NUMERICAL DYNAMIC PROGRAMMING
Kenneth L. Judd
Hoover Institution and NBER
June 28, 2006
1

Dynamic Programming
• Foundation of dynamic economic modelling
— Individual decisionmaking
— Social planners problems, Pareto efficiency
— Dynamic games
• Computational considerations
— Applies a wide range of numerical methods: Optimization, approximation, integration
— Can exploit any architecture, including high-power and high-throughput computing
Outline
• Review of Dynamic Programming
• Necessary Numerical Techniques
— Approximation
— Integration
• Numerical Dynamic Programming
2

Discrete-Time Dynamic Programming
• Objective:
E
( TX
t=1
π(x t ,u t ,t)+W (x T +1 )
)
, (12.1.1)
— X is set of states and D is the set of controls
— π(x, u, t) payoffs inperiodt, forx ∈ X at the beginning of period t, andcontrolu ∈ D is
applied in period t.
— D(x, t) ⊆ D: controls which are feasible in state x at time t.
— F (A; x, u, t) :probability that x t+1 ∈ A ⊂ X conditional on time t control and state
• Value function definition
( TX
V (x, t) ≡ sup E
U(x,t) s=t
π(x s ,u s ,s)+W (x T +1 )|x t = x
)
. (12.1.2)
• Bellman equation
V (x, t) =
sup
u∈D(x,t)
π(x, u, t)+E {V (x t+1 ,t+1)|x t = x, u t = u} (12.1.3)
• Existence: boundedness of π is sufficient
3

Autonomous, Infinite-Horizon Problem:
• Objective:
max
u t
( ∞
)
X
E β t π(x t ,u t )
t=1
(12.1.1)
• Value function definition: if U(x) is set of all feasible strategies starting at x.
( ∞ )
X
V (x) ≡ sup E β t π(x t ,u t )
U(x)
¯ x0 = x , (12.1.8)
• Bellman equation for V (x)
t=0
V (x) = sup
u∈D(x)
π(x, u)+βE © V (x + )|x, u ª ≡ (TV)(x), (12.1.9)
• Optimal policy function, U(x), ifitexists,isdefined by
U(x) ∈ arg max π(x, u)+βE© V (x + )|x, u ª
u∈D(x)
• Standard existence theorem: If X is compact, β

Deterministic Growth Example
• Problem:
— Euler equation:
V (k 0 )=max ct
P ∞
t=0 βt u(c t ),
k t+1 = F (k t ) − c t
k 0 given
u 0 (c t )=βu 0 (c t+1 )F 0 (k t+1 )
(12.1.12)
— Bellman equation
V (k) =max u(c)+βV (F (k) − c). (12.1.13)
c
— Solution to (12.1.12) is a policy function C(k) and a value function V (k) satisfying
0=u 0 (C(k))F 0 (k) − V 0 (k) (12.1.15)
V (k)=u(C(k)) + βV (F (k) − C(k)) (12.1.16)
• (12.1.16) defines the value of an arbitrary policy function C(k), not just for the optimal C(k).
• The pair (12.1.15) and (12.1.16)
— expresses the value function given a policy, and
— a first-order condition for optimality.
5

Stochastic Growth Accumulation
• Problem:
( ∞
X
)
V (k, θ) =max
c t , t
E
t=0
β t u(c t )
k t+1 = F (k t ,θ t ) − c t
θ t+1 = g(θ t ,ε t )
ε t : i.i.d. random variable
k 0 = k, θ 0 = θ.
• State variables:
— k: productive capital stock, endogenous
— θ: productivity state, exogenous
• The dynamic programming formulation is
V (k, θ) =max
c
u(c)+βE{V (F (k, θ) − c, θ + )|θ} (12.1.21)
θ + = g(θ, ε)
• The control law c = C(k, θ) satisfies the first-order conditions
0=u c (C(k, θ)) − βE{u c (C(k + ,θ + ))F k (k + ,θ + ) | θ}, (12.1.23)
where
k + ≡ F (k, L(k, θ),θ) − C(k, θ),
6

Discrete State Space Problems
• State space X = {x i ,i=1, ··· ,n}
• Controls D = {u i |i =1, ..., m}
• qij t (u) =Pr(x t+1 = x j |x t = x i ,u t = u)
• Q t (u) = ¡ qij t (u)¢ : Markov transition matrix at t if u i,j t = u.
Value Function Iteration: Discrete-State Problems
• State space X = {x i ,i=1, ··· ,n} and controls D = {u i |i =1, ..., m}
• Terminal value:
V T +1
i
= W (x i ),i=1, ··· ,n.
• Bellman equation: time t value function is
V t
i
=max
u
[π(x i ,u,t)+β
nX
j=1
q t ij(u) V t+1
j ], i=1, ··· ,n
• Bellman equation can be directly implemented - called value function iteration. Only choice for
finite T .
7

• Infinite-horizon problems
— Bellman equation is now a simultaneous set of equations for V i values:
⎡
⎤
nX
V i =max ⎣π(x i ,u)+β q ij (u) V j
⎦ ,i=1, ··· ,n
u
— Value function iteration is
V k+1
i
U k+1
i
=max
u
⎡
=argmax
u
⎣π(x i ,u)+β
⎡
j=1
nX
j=1
⎣π(x i ,u)+β
q ij (u) V k
j
nX
j=1
— Can use value function iteration with arbitrary V 0
i
— Error is given by contraction mapping property:
°
°V k − V ∗° ° ≤ 1
1 − β
⎤
q ij (u) V k
j
⎦ ,i=1, ··· ,n
⎤
⎦ ,i=1, ··· ,n
and iterate k →∞.
°
°V k+1 − V k° °
— Stopping rule: continue until ° °V k − V ∗° °

Policy Iteration (a.k.a. Howard improvement)
• Value function iteration is a slow process
— Linear convergence at rate β
— Convergence is particularly slow if β is close to 1.
• Policy iteration is faster
— Current guess:
V k
i ,i=1, ··· ,n.
— Iteration: compute optimal policy today if V k is value tomorrow:
⎡
⎤
nX
Ui
k+1 =argmax ⎣π(x i ,u)+β q ij (u) Vj
k ⎦ ,i=1, ··· ,n,
u
— Compute the value function if the policy U k+1 is used forever, which is solution to the linear
system
Vi k+1 = π ¡ ¢
nX
x i ,Ui
k+1 + β q ij (Ui k+1 ) Vj
k+1 ,i=1, ··· ,n,
j=1
— Policy iteration depends on only monotonicity
∗ If initial guess is above or below solution then policy iteration is between truth and value
function iterate
∗ Works well even for β close to 1.
9
j=1

Linear Programming Approach
• If D is finite, we can reformulate dynamic programming as a linear programming problem.
• (12.3.4) is equivalent to the linear program
P
min n
Vi i=1 V i
s.t. V i ≥ π(x i ,u)+β P n
j=1 q ij(u)V j , ∀i, u ∈ D,
• Computational considerations
(12.4.10)
— (12.4.10) may be a large problem
— Trick and Zin (1997) pursued an acceleration approach with success.
— Recent work by Daniela Pucci de Farias and Ben van Roy has revived interest.
Continuous states: Discretization
• Method:
— “Replace” continuous X with a finite X ∗ = {x i ,i=1, ··· ,n} ⊂ X
— Proceedwithafinite-state method.
• Problems:
— Sometimes need to alter space of controls to assure landing on an x in X.
— A fine discretization often necessary to get accurate approximations
10

Continuous Methods for Continuous-State Problems
• Basic Bellman equation:
V (x) = max
u∈D(x) π(u, x)+βE{V (x+ )|x, u)} ≡ (TV)(x). (12.7.1)
— Discretization essentially approximates V withastepfunction
— Approximation theory provides better methods to approximate continuous functions.
• General Task
— Choose a finite-dimensional parameterization
V (x) . = ˆV (x; a), a∈ R m (12.7.2)
and a finite number of states
X = {x 1 ,x 2 , ··· ,x n }, (12.7.3)
— Find coefficients a ∈ R m such that ˆV (x; a) “approximately” satisfies the Bellman equation.
11

General Parametric Approach: Approximating T
• For each x j , (TV)(x j ) is defined by
v j =(TV)(x j )= max
u∈D(x j ) π(u, x j)+β
Z
ˆV (x + ; a)dF (x + |x j ,u) (12.7.5)
• In practice, we compute the approximation ˆT
v j =(ˆTV)(x j ) . =(TV)(x j )
— Integration step: for ω j and x j for some numerical quadrature formula
Z
E{V (x + ; a)|x j ,u)}= ˆV (x + ; a)dF (x + |x j ,u)
Z
= ˆV (g(x j ,u,ε); a)dF (ε)
.
= X ω ˆV (g(x j ,u,ε ); a)
— Maximization step: for x i ∈ X, evaluate
— Fitting step:
v i =(T ˆV )(x i )
∗ Data: (v i ,x i ),i=1, ··· ,n
∗ Objective: find an a ∈ R m such that ˆV (x; a) best fits the data
∗ Methods: determined by ˆV (x; a)
12

Approximation Methods
• General Objective: Given data about f(x) construct simpler g(x) approximating f(x).
• Questions:
— What data should be produced and used?
— What family of “simpler” functions should be used?
— What notion of approximation do we use?
• Comparisons with statistical regression
— Both approximate an unknown function and use a finite amount of data
— Statistical data is noisy but we assume data errors are small
— Nature produces data for statistical analysis but we produce the data in function approximation
13

Interpolation Methods
• Interpolation: find g (x) from an n-D family of functions to exactly fit n data items
• Lagrange polynomial interpolation
— Data: (x i ,y i ) ,i=1, .., n.
— Objective: Find a polynomial of degree n − 1, p n (x), which agrees with the data, i.e.,
y i = f(x i ),i=1, .., n
— Result: If the x i are distinct, there is a unique interpolating polynomial
• Does p n (x) converge to f (x) as we use more points? Consider f(x) = 1
1+x 2 ,x i uniform on [−5, 5]
Figure 1:
14

• Hermite polynomial interpolation
— Data: (x i ,y i ,y 0 i) ,i=1, .., n.
— Objective: Find a polynomial of degree 2n − 1, p(x), which agrees with the data, i.e.,
y i =p(x i ),i=1, .., n
yi 0 =p 0 (x i ),i=1, .., n
— Result: If the x i are distinct, there is a unique interpolating polynomial
• Least squares approximation
— Data: A function, f(x).
— Objective: Find a function g(x) from a class G that best approximates f(x), i.e.,
g =argmax
g∈G
kf − gk2
15

Orthogonal polynomials
• General orthogonal polynomials
— Space: polynomials over domain D
— weighting function: w(x) > 0
— Inner product: hf,gi = R D f(x)g(x)w(x)dx
— Definition: {φ i } is a family of orthogonal polynomials w.r.t w (x) iff
­
φi ,φ j
® =0,i6= j
— We like to compute orthogonal polynomials using recurrence formulas
φ 0 (x)=1
φ 1 (x)=x
φ k+1 (x)=(a k+1 x + b k ) φ k (x)+c k+1 φ k−1 (x)
16

• Chebyshev polynomials
— [a, b] =[−1, 1] and w(x) = ¡ 1 − x 2¢ −1/2
— T n (x) =cos(n cos −1 x)
T 0 (x) =1
T 1 (x) =x
T n+1 (x) =2xT n (x) − T n−1 (x),
• General Orthogonal Polynomials
— Few problems have the specific intervals and weights used in definitions
— One must adapt interval through linear COV: If compact interval [a, b] is mapped to [−1, 1] by
y = −1+2 x − a
b − a
¡ ¢ ¡
then φ i −1+2
x−a
b−a are orthogonal over x ∈ [a, b] with respect to w −1+2
x−a
b−a¢
iff φi (y) are
orthogonal over y ∈ [−1, 1] w.r.t. w (y)
17

Regression
• Data: (x i ,y i ) ,i=1, .., n.
• Objective: Find a function f(x; β) with β ∈ R m .
, m ≤ n, with y i = f(xi ),i=1, .., n.
• Least Squares regression:
Chebyshev Regression
• Chebyshev Regression Data:
X
min (yi − f (x i ; β)) 2
β∈R m
• (x i ,y i ) ,i=1, .., n > m, x i are the n zeroes of T n (x) adapted to [a, b]
• Chebyshev Interpolation Data:
(x i ,y i ) ,i=1, .., n = m, x i are the n zeroes of T n (x)adapted to [a, b]
18

Algorithm 6.4: Chebyshev Approximation Algorithm in R 1
• Objective: Given f(x) defined on [a, b], find its Chebyshev polynomial approximation p(x)
• Step 1: Compute the m ≥ n +1Chebyshev interpolation nodes on [−1, 1]:
µ 2k − 1
z k = −cos
2m π ,k=1, ··· ,m.
• Step 2: Adjust nodes to [a, b] interval:
µ b − a
x k =(z k +1) + a, k =1, ..., m.
2
• Step 3: Evaluate f at approximation nodes:
w k = f(x k ) ,k=1, ··· ,m.
• Step 4: Compute Chebyshev coefficients, a i ,i=0, ··· ,n:
P m
k=1
a i =
w kT i (z k )
P m
k=1 T i(z k ) 2
to arrive at approximation of f(x, y) on [a, b]:
nX
µ
p(x) = a i T i 2 x − a
b − a − 1
i=0
19

Minmax Approximation
• Data: (x i ,y i ) ,i=1, .., n.
• Objective: L ∞ fit
• Problem: Difficult to compute
min max
β∈R m i
ky i − f (x i ; β)k
• Chebyshevminmaxproperty
Theorem 1 Suppose f :[−1, 1] → R is C k for some k ≥ 1, andletI n be the degree n polynomial
interpolation of f based at the zeroes of T n (x). Then
µ
2
k f − I n k ∞ ≤
π log(n +1)+1
• Chebyshev interpolation:
×
(n − k)!
n!
³ µ π
´k b − a k
k f (k) k ∞
2 2
— converges in L ∞
— essentially achieves minmax approximation
— easy to compute
— does not approximate f 0 20

Splines
Definition 2 Afunctions(x) on [a, b] is a spline of order n iff
1. s is C n−2 on [a, b], and
2. there is a grid of points (called nodes) a = x 0

• Conditions:
— 2n interpolation and continuity conditions:
y i =a i + b i x i + c i x 2 i + d i x 3 i ,
i =1,.,n
y i =a i+1 + b i+1 x i + c i+1 x 2 i + d i+1 x 3 i ,
i =0,.,n− 1
— 2n − 2 conditions from C 2 at the interior: for i =1, ···n − 1,
b i +2c i x i +3d i x 2 i =b i+1 +2c i+1 x i +3d i+1 x 2 i
2c i +6d i x i =2c i+1 +6d i+1 x i
— Equations (1—4) are 4n − 2 linear equations in 4n unknown parameters, a, b, c, and d.
— construct 2 side conditions:
∗ natural spline: s 0 (x 0 )=0=s 0 (x n ); it minimizes total curvature, R x n
x 0
s 00 (x) 2 dx, among
solutions to (1-4).
∗ Hermite spline: s 0 (x 0 )=y0 0 and s 0 (x n )=yn 0 (assumes extra data)
∗ Secant Hermite spline: s 0 (x 0 )=(s(x 1 )−s(x 0 ))/(x 1 −x 0 ) and s 0 (x n )=(s(x n )−s(x n−1 ))/(x n −
x n−1 ).
∗ not-a-knot: choosej = i 1 ,i 2 , such that i 1 +1

• Shape-preservation
— Concave (monotone) data may lead to nonconcave (nonmonotone) approximations.
— Example
• Schumaker Procedure:
1. Take level (and maybe slope) data at nodes x i
2. Add intermediate nodes z + i ∈ [x i ,x i+1 ]
3. Run quadratic spline with nodes at the x and z nodes which intepolate data and preserves
shape.
4. Schumaker formulas tell one how to choose the z and spline coefficients (see book and correction
at book’s website)
• Many other procedures exist for one-dimensional problems, but few procedures exist for twodimensional
problems
23

• Spline summary:
— Evaluation is cheap
∗ Splines are locally low-order polynomial.
∗ Can choose intervals so that finding which [x i ,x i+1 ] contains a specific x is easy.
∗ Finding enclosing interval for general x i sequence requires at most dlog 2 ne comparisons
— Good fits even for functions with discontinuous or large higher-order derivatives. E.g., quality
of cubic splines depends only on f (4) (x), not f (5) (x).
— Can use splines to preserve shape conditions
24

Multidimensional approximation methods
• Lagrange Interpolation
— Data: D ≡ {(x i ,z i )} N i=1 ⊂ Rn+m ,wherex i ∈ R n and z i ∈ R m
— Objective: find f : R n → R m such that z i = f(x i ).
— Need to choose nodes carefully.
— Task: Find combinations of interpolation nodes and spanning functions to produce a nonsingular
(well-conditioned) interpolation matrix.
25

Tensor products
• General Approach:
— If A and B are sets of functions over x ∈ R n , y ∈ R m , their tensor product is
A ⊗ B = {ϕ(x)ψ(y) | ϕ ∈ A, ψ ∈ B}.
— Given a basis for functions of x i , Φ i = {ϕ i k (x i)} ∞ k=0 , the n-fold tensor product basis for functions
of (x 1 ,x 2 ,...,x n ) is
( nY
)
Φ = ϕ i k i
(x i ) | k i =0, 1, ··· ,i=1,...,n
i=1
• Orthogonal polynomials and Least-square approximation
— Suppose Φ i are orthogonal with respect to w i (x i ) over [a i ,b i ]
— Least squares approximation of f(x 1 , ··· ,x n ) in Φ is
X hϕ, fi
hϕ, ϕi ϕ,
where the product weighting function
defines h·, ·i over D = Q i [a i,b i ] in
ϕ∈Φ
W (x 1 ,x 2 , ··· ,x n )=
hf(x),g(x)i =
Z
D
nY
i=1
w i (x i )
f(x)g(x)W (x)dx.
26

Algorithm 6.4: Chebyshev Approximation Algorithm in R 2
• Objective: Given f(x, y) defined on [a, b] × [c, d], find its Chebyshev polynomial approximation
p(x, y)
• Step 1: Compute the m ≥ n +1Chebyshev interpolation nodes on [−1, 1]:
µ 2k − 1
z k = −cos
2m π ,k=1, ··· ,m.
• Step 2: Adjust nodes to [a, b] and [c, d] intervals:
µ b − a
x k =(z k +1) + a, k =1, ..., m.
2
µ d − c
y k =(z k +1) + c, k =1, ..., m.
2
• Step 3: Evaluate f at approximation nodes:
w k, = f(x k ,y ) ,k=1, ··· ,m. , =1, ··· ,m.
• Step 4: Compute Chebyshev coefficients, a ij ,i,j =0, ··· ,n:
P m P m
k=1 =1
a ij =
w k,T i (z k )T j (z )
( P m
k=1 T i(z k ) 2 )( P m
=1 T j(z ) 2 )
to arrive at approximation of f(x, y) on [a, b] × [c, d]:
nX nX
µ
p(x, y) = a ij T i 2 x − a µ
b − a − 1 T j 2 y − c
d − c − 1
i=0
j=0
27

Multidimensional Splines
• B-splines: Multidimensional versions of splines can be constructed through tensor products; here
B-splines would be useful.
• Summary
— Tensor products directly extend one-dimensional methods to n dimensions
— Curse of dimensionality often makes tensor products impractical
Complete polynomials
• Taylor’s theorem for R n produces the approximation
f(x) . =f(x 0 )+ P n
i=1
+ 1 2
P n
i 1 =1
P n
i 2 =1
∂f
∂x i
(x 0 )(x i − x 0 i )
∂ 2 f
∂x i1 ∂x ik
(x 0 )(x i1 − x 0 i 1
)(x ik − x 0 i k
)+...
— For k =1, Taylor’s theorem for n dimensions used the linear functions P1 n ≡ {1,x 1 ,x 2 , ··· ,x n }
— For k =2, Taylor’s theorem uses P2 n ≡ P1 n ∪ {x 2 1, ··· ,x 2 n,x 1 x 2 ,x 1 x 3 , ··· ,x n−1 x n }.
• In general, the kth degree expansion uses the complete set of polynomials of total degree k in n
variables.
nX
Pk n ≡ {x i 1
1 ···x i n n | i ≤ k, 0 ≤ i 1 , ··· ,i n }
=1
28

• Complete orthogonal basis includes only terms with total degree k or less.
• Sizes of alternative bases
degree k Pk n Tensor Prod.
2 1+n + n(n +1)/2 3 n
3 1+n + n(n+1)
2
+ n 2 + n(n−1)(n−2)
6
4 n
— Complete polynomial bases contains fewer elements than tensor products.
— Asymptotically, complete polynomial bases are as good as tensor products.
— For smooth n-dimensional functions, complete polynomials are more efficient approximations
• Construction
— Compute tensor product approximation, as in Algorithm 6.4
— Drop terms not in complete polynomial basis (or, just compute coefficients for polynomials in
complete basis).
— Complete polynomial version is faster to compute since it involves fewer terms
29

Integration
• Most integrals cannot be evaluated analytically
• Integrals frequently arise in economics
— Expected utility and discounted utility and profits over a long horizon
— Bayesian posterior
— Solution methods for dynamic economic models
Gaussian Formulas
• All integration formulas choose quadrature nodes x i ∈ [a, b] and quadrature weights ω i :
Z b
f(x) dx =
. nX
ω i f(x i ) (7.2.1)
a
— Newton-Cotes (trapezoid, Simpson, etc.) use arbitrary x i
— Gaussian quadrature uses good choices of x i nodes and ω i weights.
• Exact quadrature formulas:
— Let F k bethespaceofdegreek polynomials
— A quadrature formula is exact of degree k if it correctly integrates each function in F k
— Gaussian quadrature formulas use n points and are exact of degree 2n − 1
i=1
30

Theorem 3 Suppose that {ϕ k (x)} ∞ k=0
is an orthonormal family of polynomials with respect to w(x)
on [a, b]. Then there are x i nodes and weights ω i such that a

Gauss-Chebyshev Quadrature
• Domain: [−1, 1]
• Weight: (1 − x 2 ) −1/2
• Formula: Z 1
f(x)(1 − x 2 ) −1/2 dx = π
−1 n
for some ξ ∈ [−1, 1], with quadrature nodes
Arbitrary Domains
• Want to approximate R b
a
x i = cos
nX
f(x i )+
i=1
π f (2n) (ξ)
2 2n−1 (2n)!
(7.2.4)
µ 2i − 1
2n π
, i =1, ..., n. (7.2.5)
f(x) dx for different range, and/or no weight function
— Linear change of variables x = −1+2(y − a)(b − a)
— Multiply the integrand by (1 − x 2 ) ± 1/2 (1 − x 2 ) 1/2 .
Z b
f(y) dy = b − a Z 1
µ (x +1)(b − a)
f
2
2
a
−1
¡ ¢ 1 − x
2 1/2
+ a
dx
(1 − x 2 1/2
)
— Gauss-Chebyshev quadrature uses the x i Gauss-Chebyshev nodes over [−1, 1]
Z b
f(y) dy = . π(b − a)
nX
µ (xi +1)(b − a) ¡1 ¢
f
+ a − x
2 1/2
2n
2
i
a
i=1
32

Gauss-Hermite Quadrature
• Domain is [−∞, ∞] and weight is e −x2
• Formula: for some ξ ∈ (−∞, ∞).
Z ∞
−∞
f(x)e −x2 dx =
nX
i=1
ω i f(x i )+ n!√ π
· f (2n) (ξ)
2 n (2n)!
N x i ω i
2 0.7071067811 0.8862269254
3 0.1224744871(1) 0.2954089751
0.0000000000 0.1181635900(1)
N x i ω i
7 0.2651961356(1) 0.9717812450(−3)
0.1673551628(1) 0.5451558281(−1)
0.8162878828 0.4256072526
0.0000000000 0.8102646175
• Normal Random Variables
— Y is distributed N(µ, σ 2 ). Expectation is integration.
— Use Gauss-Hermite quadrature: Linear COV x =(y − µ)/ √ 2 σ implies
E{f(Y )}=
Z ∞
−∞
.
=π −1 2
f(y)e −(y−µ)2 /(2σ 2) dy =
nX
ω i f( √ 2 σx i + µ)
i=1
Z ∞
−∞
f( √ 2 σx+ µ)e −x2 √
2 σdx
where the ω i and x i are the Gauss-Hermite quadrature weights and nodes over [−∞, ∞].
33

Multidimensional Integration
• Most economic problems have several dimensions
— Multiple assets
— Multiple error terms
• Multidimensional integrals are much more difficult
— Simple methods suffer from curse of dimensionality
— There are methods which avoid curse of dimensionality
34

Product Rules
• Build product rules from one-dimension rules
• Let x i,ω i, i =1, ··· ,m, be one-dimensional quadrature points and weights in dimension from
a Newton-Cotes rule or the Gauss-Legendre rule.
• The product rule
Z
[−1,1] d f(x)dx . =
mX mX
···
i 1 =1 i d =1
ω 1 i 1
ω 2 i 2 ···ω d i d
f(x 1 i 1
,x 2 i 2
, ··· ,x d i d
)
• Gaussian structure prevails
— Suppose w (x) is weighting function in dimension
— Define the d-dimensional weighting function.
dY
W (x) ≡ W (x 1 , ··· ,x d )= w (x )
— Product Gaussian rules are based on product orthogonal polynomials.
• Curse of dimensionality:
— m d functional evaluations is m d for a d-dimensional problem with m points in each direction.
— Problem worse for Newton-Cotes rules which are less accurate in R 1 .
=1
35

General Parametric Approach: Approximating T
• For each x j , (TV)(x j ) is defined by
v j =(TV)(x j )= max
u∈D(x j ) π(u, x j)+β
Z
ˆV (x + ; a)dF (x + |x j ,u) (12.7.5)
• In practice, we compute the approximation ˆT
v j =(ˆTV)(x j ) . =(TV)(x j )
— Integration step: for ω j and x j for some numerical quadrature formula
Z
E{V (x + ; a)|x j ,u)}= ˆV (x + ; a)dF (x + |x j ,u)
Z
= ˆV (g(x j ,u,ε); a)dF (ε)
.
= X ω ˆV (g(x j ,u,ε ); a)
— Maximization step: for x i ∈ X, evaluate
v i =(T ˆV )(x i )
∗ Hot starts
∗ Concave stopping rules
— Fitting step:
∗ Data: (v i ,x i ),i=1, ··· ,n
∗ Objective: find an a ∈ R m such that ˆV (x; a) best fits the data
∗ Methods: determined by ˆV (x; a)
36

Approximating T with Hermite Data
• Conventional methods just generate data on V (x j ):
Z
v j = max π(u, x j)+β ˆV (x + ; a)dF (x + |x j ,u) (12.7.5)
u∈D(x j )
• Envelope theorem:
— If solution u is interior,
Z
vj 0 = π x (u, x j )+β
ˆV (x + ; a)dF x (x + |x j ,u)
— If solution u is on boundary
Z
vj 0 = µ + π x (u, x j )+β
ˆV (x + ; a)dF x (x + |x j ,u)
where µ is a Kuhn-Tucker multiplier
• Since computing vj 0 is cheap, we should include it in data:
— Data: (v i ,vi 0,x i), i=1, ··· ,n
— Objective: find an a ∈ R m such that ˆV (x; a) best fits Hermite data
— Methods: determined by ˆV (x; a)
37

General Parametric Approach: Value Function Iteration
• Comparison with discretization
guess a−→ ˆV (x; a)
−→ (v i ,x i ),i=1, ··· ,n
−→ new a
— This procedure examines only a finite number of points, but does not assume that future points
lie in same finite set.
— Our choices for the x i are guided by systematic numerical considerations.
• Synergies
— Smooth interpolation schemes allow us to use Newton’s method in the maximization step.
— They also make it easier to evaluate the integral in (12.7.5).
• Finite-horizon problems
— Value function iteration is only possible procedure since V (x, t) depends on time t.
— Begin with terminal value function, V (x, T )
— Compute approximations for each V (x, t), t = T − 1,T − 2, etc.
38

Algorithm 12.5: Parametric Dynamic Programming
with Value Function Iteration
Objective: Solve the Bellman equation, (12.7.1).
Step 0: Choose functional form for ˆV (x; a), and choose
the approximation grid, X = {x 1 , ..., x n }.
Make initial guess ˆV (x; a 0 ), and choose stopping
criterion >0.
Step 1: Maximization step: Compute
v j =(T ˆV (·; a i ))(x j ) for all x j ∈ X.
Step 2: Fitting step: Using the appropriate approximation
method, compute the a i+1 ∈ R m such that
ˆV (x; a i+1 ) approximates the (v i ,x i ) data.
Step 3: If k ˆV (x; a i ) − ˆV (x; a i+1 ) k< , STOP; else go to step 1.
39

• Convergence
— T is a contraction mapping
— ˆT may be neither monotonic nor a contraction
• Shape problems
— An instructive example
Figure 2:
— Shape problems may become worse with value function iteration
— Shape-preserving approximation will avoid these instabilities
40

Summary:
• Discretization methods
— Easy to implement
— Numerically stable
— Amenable to many accelerations
— Poor approximation to continuous problems
• Continuous approximation methods
— Can exploit smoothness in problems
— Possible numerical instabilities
— Acceleration is less possible
41