Linear Programming Approach to Deterministic Infinite Horizon ...

SIAM J. CONTROL OPTIM. 

Vol. 48, No. 4, pp. 2480–2512 

c○ 2009 Society for Industrial and Applied Mathematics 

LINEAR PROGRAMMING APPROACH TO DETERMINISTIC 

INFINITE HORIZON OPTIMAL CONTROL PROBLEMS WITH 

DISCOUNTING ∗ 

VLADIMIR GAITSGORY † AND MARC QUINCAMPOIX ‡ 

Abstract. We investigate relationships between the deterministic infinite time horizon optimal 

control problem with discounting, in which the state trajectories remain in a given compact set Y , 

and a certain infinite dimensional linear programming (IDLP) problem. We introduce the problem 

dual with respect to this IDLP problem and obtain some duality results. We construct necessary 

and sufficient optimality conditions for the optimal control problem under consideration, and we give 

a characterization of the viability kernel of Y . We also indicate how one can use finite dimensional 

approximations of the IDLP problem and its dual for construction of near optimal feedback controls. 

The construction is illustrated with a numerical example. 

Key words. optimal control problems with discounting, long run average optimal control, 

occupational measures, averaging, linear programming, duality, viability kernels, numerical solution 

AMS subject classifications. 34E15, 34C29, 34A60, 93C70 

DOI. 10.1137/070696209 

1. Introduction and preliminaries. It is well known that the dynamics of a 

nonlinear stochastic control system has a linear representation through the dynamics 

of the corresponding state-control probability distributions. A different (but related) 

idea of “linearizing” nonlinear optimal control problems can be realized through reformulating 

these as optimization problems on spaces of occupational measures, which, 

under mild conditions, can be shown to be “equivalent” (or “asymptotically equivalent”) 

to certain infinite dimensional (ID) linear programming (LP) problems. This 

idea is applicable to both stochastic and deterministic settings. It is based on the fact 

that the occupational measures generated by admissible controls and the corresponding 

solutions of a nonlinear system satisfy certain linear equations representing the 

system’s dynamics in a relaxed integral form. 

Fundamental results that justify the use of LP formulations in various problems of 

optimal control of stochastic systems have been obtained in [11], [24], [34], [39], [54], 

[55]. Important advances in the development of IDLP formulations in deterministic 

optimal control problems considered on finite time intervals have been made in [35], 

[42], [51], [58] (and in some earlier papers mentioned therein). Also various aspects of 

the LP approach to deterministic problems of optimal control with long run average 

criteria were studied in [22] and [30] (important related developments can be found 

in [21] and [31]). 

This paper is devoted to the development of the LP approach to the deterministic 

infinite horizon optimal control problem with discounting, in which the state trajec- 

∗ Received by the editors July 4, 2007; accepted for publication (in revised form) May 15, 2009; 

published electronically August 14, 2009. 

http://www.siam.org/journals/sicon/48-4/69620.html 

† Center for Industrial and Applied Mathematics, University of South Australia, Mawson Lakes, 

SA 5095, Australia (v.gaitsgory@unisa.edu.au). This author’s work was partially supported by the 

Australian Research Council Discovery grants DP0664330 and DP0986696, and by Linkage International 

grants LX0560049 and LX0881972. 

‡ LaboratoiredeMathématiques, unité CNRS UMR6205, Université de Bretagne Occidentale, 6 

Avenue Victor Le Gorgeu, 29200 Brest, France (marc.quincampoix@univ-brest.fr). This author’s 

work was partially supported by Linkage International grant LX0560049. 

2480 

Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.

LP IN OPTIMAL CONTROL PROBLEMS WITH DISCOUNTING 2481 

tories remain in a compact set Y (a state constraint). We establish that the optimal 

value of this problem coincides with the optimal value of a certain IDLP problem, 

the feasible set of the latter coinciding with the convex hull of the set of discounted 

occupational measures generated by the control system, and we show that this IDLP 

problem and its dual can be used for construction of necessary and sufficient optimality 

conditions and for a characterization of the viability kernel of Y . We also indicate 

a way of how one can use finite dimensional approximations of the IDLP problem and 

its dual for construction of near optimal feedback controls (the construction being 

illustrated with a numerical example). 

Note that infinite horizon problems of optimal control arise in many applications 

(in engineering, economics, environmental modeling, etc.). They have been studied 

intensively(see,e.g.,[1],[3],[4],[7],[8],[10],[11],[15],[17],[20],[23],[24],[32], 

[34], [36], [39], [40], [41], [44], [47], [50], [56], [60]), and the present paper aims at 

contributing to this important line of research. 

The control system we will be dealing with is of the form 

(1.1) y ′ (t) =f(y(t),u(t)), t≥ 0, 

where the function f(·) :R m × U ↦→ R m is continuous in (y, u) and satisfies Lipschitz 

conditions in y uniformly with respect to u. The controls are Lebesgue measurable 

functions u(·) :[0,S] ↦→ U or u(·) :[0, +∞) ↦→ U (depending on whether the system 

is considered on the finite time interval [0,S] or on the infinite time interval [0, +∞)), 

where U is a compact metric space. The sets of these controls are denoted as U S and 

U, respectively. A solution of the system (1.1) obtained with a control u(·) andwith 

the initial condition y(0) = y 0 will be denoted as y(t, y 0 ,u(·)). 

Let Y be a nonempty compact subset of R m . We will be considering the solutions 

of the system (1.1), which satisfy the state constraint 

(1.2) y(t, y 0 ,u(·)) ∈ Y, 

and we will denote by US Y (y 0) ⊂U S and by U Y (y 0 ) ⊂U the sets of controls such 

that (1.2) is satisfied for all t ∈ [0,S]andforallt ∈ [0, ∞), respectively. Note that 

the set Y is called viable if U Y (y 0 ) ≠ ∅ for all y 0 ∈ Y (see [5]). 

Let us consider the optimal control problem 

(1.3) inf 

u(·)∈U Y (y 0) 

∫ +∞ 

0 

e −Ct g(y(t, y 0 ,u(·)),u(t))dt def 

= V Y C (y 0), 

where g : R m × U ↦→ R is continuous and satisfies Lipschitz conditions in y uniformly 

with respect to u and where C>0 is the discount factor. We will be interested in 

establishing connections between this problem and the problem 

∫ 

(1.4) inf 

γ∈W (C,y 0) 

where 

(1.5) 

W (C, y 0 ) def 

= 

{ 

γ ∈P(Y × U) : 

Y ×U 

∫ 

Y ×U 

g(y, u)γ(dy, du) def 

= g ∗ (C, y 0 ), 

(∇φ(y) T f(y, u)+C(φ(y 0 ) − φ(y)))γ(dy, du) 

=0 ∀φ ∈ C 1 }, 


2482 VLADIMIR GAITSGORY AND MARC QUINCAMPOIX 

with P(Y ×U) standing for the space of probability measures defined on Borel subsets 

of Y × U. Note that problem (1.4) is of IDLP since its objective function and its constraints 

are linear in γ (see, e.g., [2]). Note that the key element of our consideration 

is the fact that the discounted occupational measures generated by the solutions of 

system (1.1) satisfy the constraints defining W (C, y 0 ) (see Proposition 2.2). 

Along with (1.3), we will also be considering the optimal control problem 

(1.6) 

∫ 

1 

S 

S inf inf g(y(t, y 0 ,u(·)),u(t))dt def 

= G S . 

y 0∈Y u(·)∈US Y (y0) 0 

In [22] and [30] it has been established that this problem (considered with S →∞) 

is closely related to the IDLP problem 

∫ 

(1.7) inf g(y, u)γ(dy, du) def 

= g ∗ , 

γ∈W 

where 

(1.8) W def 

= 

{ 

γ ∈P(Y × U) : 

Y ×U 

∫ 

Y ×U 

∇φ(y) T f(y, u)γ(dy, du) =0 ∀φ ∈ C 1 }. 

The argument used in [22] and [30] was based on results from stochastic control 

theory (mainly, on the fundamental result characterizing the set of one-dimensional 

marginal stationary distributions of a control martingale problem obtained in [54]; 

see also [11] and [39]). We will show that some of the results of [22] and [30] that 

establish relationships between (1.6) and (1.7) can be obtained on the basis of results 

establishing relationships between (1.3) and (1.4) (by letting the discount factor tend 

to zero). 

The rest of the paper is organized as follows. In section 2, we reformulate the 

optimal control problems (1.3) and (1.6) in terms of occupational measures, and we 

prove some preliminary results relating these problems with the IDLP problems (1.4) 

and (1.7). In section 3, we introduce the problem dual to (1.4) and obtain duality 

results. In section 4, we use results of section 3 to establish the relationships between 

(1.3) and (1.4), and we obtain necessary and sufficient optimality conditions for (1.3). 

In section 5, we demonstrate the possibility of applying results of section 4 for a 

characterization of the viability kernel of Y . In section 6, we use results of section 

4 to establish the relationships between (1.6) and (1.7). In section 7, we discuss the 

possibility of approximation of the IDLP problem and its dual with finite dimensional 

LP problems, and we illustrate that the latter can be used for finding a near optimal 

control in (1.3) with a numerical example. In section 8, we make some conclusions 

about the obtained results. In the appendix, we give some proofs that were omitted 

in the previous consideration. 

Let us conclude this section with some comments and notation. Note, first of 

all, that the space P(Y × U) is known to be compact in weak convergence (weak ∗ ) 

topology (see, e.g., [9] or [46]). Hence, the sets W and W (C, y 0 )arecompactinthis 

topology, and a solution to problem (1.7) or problem (1.4) exists as soon as W or, 

respectively, W (C, y 0 ), is not empty. 

Let us endow the space P(Y × U) with a metric ρ, 

(1.9) ρ(γ ′ ,γ ′′ ) def 

= 

∞∑ 

j=1 

∣ ∫ 

1 ∣∣∣ 

2 

∫U×Y 

j q j (y, u)γ ′ (dy, du) − 

U×Y 

q j (y, u)γ ′′ (dy, du) 

∣ , 



for all γ ′ ,γ ′′ ∈P(Y ×U), where q j (·),j =1, 2,...,is a sequence of Lipschitz continuous 

functions which is dense in the unit ball of C(Y ×U) (the space of continuous functions 

on Y × U). Note that this metric is consistent with the weak convergence topology 

of P(Y × U). Namely, a sequence γ k ∈P(Y × U) convergestoγ ∈P(Y × U) inthis 

metric if and only if 

∫ 

∫ 

(1.10) lim q(y, u)γ k (dy, du) = q(y, u)γ(dy, du) 

k→∞ U×Y 

U×Y 

for any continuous q(·) ∈ C(Y ×U). Using this metric ρ, one can define the “distance” 

ρ(γ,Γ) between γ ∈P(Y ×U)andΓ⊂P(Y ×U), and the Hausdorff metric ρ H (Γ 1 , Γ 2 ) 

between Γ 1 ⊂P(Y × U) andΓ 2 ⊂P(Y × U), as follows: 

(1.11) 

ρ(γ,Γ) def 

= inf 

γ ′ ∈Γ ρ(γ,γ′ ) , 

{ 

} 

ρ H (Γ 1 , Γ 2 ) def 

=max sup ρ(γ,Γ 2 ), sup ρ(γ,Γ 1 ) . 

γ∈Γ 1 γ∈Γ 2 

Note that, although, by some abuse of terminology, we refer to ρ H (·, ·) asametric 

on the set of subsets of P(Y × U), it is, in fact, a semimetric on this set (since 

ρ H (Γ 1 , Γ 2 )=0isequivalenttoΓ 1 =Γ 2 if and only if Γ 1 and Γ 2 are closed). 

2. Occupational measure formulations. In this section we introduce occupational 

and discounted occupational measures. We reformulate the optimal control 

problems (1.3) and (1.6) in terms of these measures, and we establish some readily 

verifiable relationships between (1.3) and (1.4) and between (1.6) and (1.7). The 

results of this section are used in the further consideration. 

Let u(·) ∈US Y (y 0)andy(t) =y(t, y 0 ,u(·)), t ∈ [0,S]. A probability measure 

γ u(·),S ∈P(Y ×U) is called the occupational measure generated by the pair (y(·),u(·)) 

on the interval [0,S] if, for any Borel set Q ⊂ Y × U, 

(2.1) γ u(·),S (Q) = 1 S 

∫ S 

0 

1 Q (y(t),u(t))dt, 

where 1 Q (·) is the indicator function of Q. This definition is equivalent to the statement 

that the equality 

(2.2) 

∫ 

Y ×U 

q(y, u)γ u(·),S (dy, du) = 1 S 

∫ S 

0 

q(y(t),u(t))dt 

is valid for any q(·) ∈ C(Y × U). 

Let u(·) ∈U Y (y 0 )andy(t) =y(t, y 0 ,u(·)), t∈ [0, ∞). The pair (y(·),u(·)) is said 

to generate an occupational measure on the interval [0, ∞) if there exists a limit 

(2.3) lim γ def 

u(·),S = γ u(·) . 

S→∞ 

Note that γ u(·) is generated by (y(·),u(·)) on [0, ∞) if and only if 

(2.4) 

∫ 

Y ×U 

for any q(·) ∈ C(Y × U). 

∫ 

1 S 

q(y, u)γ u(·) (dy, du) = lim q(y(t),u(t))dt 

S→∞ S 0 



Let u(·) ∈U Y (y 0 )andy(t) =y(t, y 0 ,u(·)), t ∈ [0, ∞). A probability measure 

γu(·) C ∈P(Y × U) is called the discounted occupational measure generated by the pair 

(y(·),u(·)) if, for any Borel set Q ⊂ Y × U, 

∫ ∞ 

(2.5) γu(·) C (Q) =C e −Ct 1 Q (y(t),u(t))dt, 

where the latter definition is equivalent to the equality 

∫ 

∫ ∞ 

(2.6) 

q(y, u)γu(·) C (dy, du) =C e −Ct q(y(t),u(t))dt 

Y ×U 

0 

being valid for any q(·) ∈ C(Y × U). 

Proposition 2.1. If γ u(·) is generated by the pair (y(·),u(·)) on [0, ∞), then 

(2.7) lim ρ(γu(·) C ,γ C→0 

u(·)) =0. 

Proof. From the fact that γ u(·) is generated by the pair (y(·),u(·)) on [0, ∞), it 

follows that the limit in the right-hand side of (2.4) exists for any q(·) ∈ C(Y × U). 

Hence, by the Abelian theorem (see, e.g., Lemma 3.5(i) in [33]), 

∫ ∞ 

∫ 

lim C e −Ct 1 S 

q(y(t),u(t))dt = lim q(y(t),u(t))dt 

C→0 0 

S→∞ S 0 

∫ 

⇒ lim q(y, u)γu(·) C→0 

∫Y C (dy, du) = q(y, u)γ u(·) (dy, du). 

×U 

Y ×U 

The validity of the latter for any q(·) ∈ C(Y × U) is equivalent to (2.7). 

Let us introduce the following notation: 

(2.8) Γ S (y 0 ) def 

= 

⋃ 

u(·)∈U Y S (y0) {γ u(·),S }, 

(2.9) Γ(C, y 0 ) def 

= 

⋃ 

0 

u(·)∈U Y (y 0) 

Γ S 

def 

= ⋃ 

{γ C u(·) }, 

y 0∈Y 

{Γ S (y 0 )}, 

with Γ S (y 0 ) def 

= ∅ if US Y (y 0)=∅ and Γ(C, y 0 ) def 

= ∅ if U Y (y 0 )=∅. Due to (2.6) 

and (2.2), respectively, problems (1.3) and (1.6) can be rewritten in this notation as, 

respectively, 

∫ 

(2.10) inf g(y, u)γ(dy, du) =CVC Y (y 0 ) 

γ∈Γ(C,y 0) 

and 

∫ 

(2.11) inf 

γ∈Γ S 

g(y, u)γ(dy, du) =G S . 

Note that these problems are not of LP (since Γ(C, y 0 )andΓ S are not defined by 

linear constraints), and our immediate aim is to relate them to IDLP problems (1.4) 

and (1.7). 



Proposition 2.2. The following relationships are valid: 

(2.12) CV Y C (y 0 ) ≥ g ∗ (C, y 0 ); 

(2.13) ¯coΓ(C, y 0 ) ⊂ W (C, y 0 ), 

where co ¯ stands for the closed convex hull. 

Proof. Take arbitrary γ ∈ Γ(C, y 0 ). By definition, there exists u(·) ∈U Y (y 0 ) 

and y(t) def 

= y(t, y 0 ,u(·)) such that γ = γu(·) C (that is, γ is the discounted occupational 

measure generated by the pair (y(·),u(·))). Using the fact that (2.6) is valid for any 

continuous function q(y, u), one can obtain 

∫ 

∫ ∞ 

∇φ(y) T f(y, u)γu(·) C (dy, du) =C e −Ct ∇φ(y(t)) T f(y(t),u(t))dt = −Cφ(y 0 ) 

Y ×U 

0 

+ C 2 ∫ ∞ 

0 

∫ 

e −Ct φ(y(t))dt = −C (φ(y 0 ) − φ(y))γu(·) C (dy, du) 

Y ×U 

∀φ ∈ C1 

⇒ γ = γ C u(·) ∈ W (C, y 0) ⇒ Γ(C, y 0 ) ⊂ W (C, y 0 ). 

The last inclusion implies (2.12), and also it implies (2.13) (since W (C, y 0 )isconvex 

and compact). 

Proposition 2.3. The following relationships are valid: 

(2.14) lim S→∞ G S ≥ g ∗ , 

(2.15) lim 

S→∞ 

max ρ(γ,W)=0. 

γ∈ ¯coΓ S 

Proof. The proof of the proposition is contained in the corresponding part of the 

proof of Theorem 2.1(i) in [28] (see also Proposition 2 and Corollary 3 in [30]). For 

the sake of completeness, we also give a sketch of the proof below. 

Let S k ,k=1, 2,..., be such that S k →∞,andlet γ k ∈ Γ(S k ). Let also γ be a 

partial limit of {γ k }. That is, there exists a subsequence {k ′ }⊂{k} such that 

(2.16) lim 

k ′ →∞ γk′ = γ. 

By (1.10), it implies that 

∫ 

∫ 

(2.17) lim (φ ′ (y)) T f(u, y)γ k (du, dy) = 

k→∞ U×Y 

U×Y 

(φ ′ (y)) T f(u, y)γ(du, dy) 

for any φ ∈ C 1 . Also, from the fact that γ k ∈ Γ(S k ), it follows that there exist an 

initial condition y k ∈ Y and a control u k (·) ∈U Y S k (y k 0 ) such that 

∫ 

U×Y 

(φ ′ (y)) T f(u, y)γ k (du, dy) = 1 ∫ S 

k 

S k (φ ′ (y k (τ))) T f(u k (τ),y k (τ))dτ, 

0 



where y k (τ) def 

= y(t, y k 0 ,uk (·)) is the corresponding solution of system (1.1). The second 

integral in the expression above is equal to 

φ(y k (S k )) − φ(y k (0)) 

S k 

and tends to zero as S k tends to infinity (since y k (τ) ∈ Y ∀ τ ∈ [0,S k ]andY is a 

compact set). This and (2.16), (2.17) imply that 

∫ 

(φ ′ (y)) T f(u, y)γ(du, dy) = 0 ∀φ ∈ C 1 ⇒ γ ∈ W, 

U×Y 

which, in turn, implies that 

(2.18) lim S→∞ Γ S ⊂ W. 

From (2.18) it follows that (2.14) is valid (see (2.11)) and also that lim S→∞ max γ∈ΓS 

ρ(γ,W) =0. The latter implies (2.13) due to the definition of ρ (see (1.9)) and due 

to the convexity of W . 

Finally, let us establish the following straightforward relationships between g ∗ (C, y 0 ) 

and g ∗ and between W (C) def 

= ⋃ y W (C, y 0∈Y 0)andW . 

Lemma 2.4. The following relationships are valid: 

(2.19) lim C→0 inf 

y 0∈Y g∗ (C, y 0 ) ≥ g ∗ ; 

(2.20) lim C→0 W (C) ⊂ W. 

Also 

(2.21) lim 

max ρ(γ,W)=0. 

C→0 γ∈ ¯coW (C) 

Proof. Let γ l ∈ W (C l ,y 0 ),l=1, 2,..., and let C l → 0asl →∞.Letalsoγ be a 

def 

partial limit of γ l . That is, lim l′ →∞ γ l ′ = γ for some {l ′ }⊂{l}. Then, by passing to 

the limit in (1.5), one can obtain that γ ∈ W . This proves (2.20). The inequality (2.19) 

follows from (2.20). Also from (2.20) it follows that lim C→0 max γ∈W (C) ρ(γ,W)=0. 

The latter implies (2.21). 

3. Dual problem and duality relationships. In this section we introduce a 

problem dual with respect to the IDLP problem (1.4) as a problem of maximization 

over functions ψ(·) ∈ C 1 (see (3.1) below), and we establish duality relationships 

between (1.4) and (3.1). We then present results allowing one to extend the class of 

functions used in the formulations of the problems (3.1) and (1.4) from C 1 to Lip (the 

class of locally Lipschitz continuous functions ψ(·) :R m → R), and we also establish 

necessary and sufficient conditions for γ ∗ ∈ W (C, y 0 ) to be an optimal solution of (1.4) 

(based on duality-type relationships). Results of this section are used to establish the 

fact that the inequality (2.12) and the inclusion (2.13) take the form of equalities and 

to construct necessary and sufficient optimality conditions for (1.3) (see section 4). 

Let us consider the problem 

{ 

(3.1) sup μ 

∣ ∃ψ(·) ∈ } 

C1 : ∀(y, u) ∈ Y × U, 

μ ≤∇ψ(y) T = μ ⋆ (C, y 

f(y, u)+C(ψ(y 0 ) − ψ(y)) + g(y, u) 

0 ), 



which we will refer to as dual with respect to (1.4) (see section 8, where we discuss how 

problem (3.1) can be constructed as a “standard” LP dual). Note that, if W (C, y 0 ) ≠ 

∅, then, for any γ ∈ W (C, y 0 ), from the fact that the pair (μ, ψ(·)) satisfies the 

inequality 

μ ≤∇ψ(y) T f(y, u)+C(ψ(y 0 ) − ψ(y)) + g(y, u) ∀(y, u) ∈ Y × U, 

it follows that μ ≤ ∫ g(y, u)γ(dy, du). Hence, the optimal values of (1.4) and its 

Y ×U 

dual (3.1) satisfy the inequality 

(3.2) μ ∗ (C, y 0 ) ≤ g ∗ (C, y 0 ). 

As follows from the theorem stated below, the inequality (3.2) turns into the equality 

(that is, there is no duality gap) if and only if W (C, y 0 )isnotempty. 

Theorem 3.1. (i) The optimal value of problem (3.1) is bounded (that is, 

μ ∗ (C, y 0 ) < ∞) if and only if the set W (C, y 0 ) is not empty; (ii) if the optimal 

value of problem (3.1) is bounded, then 

(3.3) μ ∗ (C, y 0 )=g ∗ (C, y 0 ); 

(iii) the optimal value of of problem (3.1) is unbounded (that is, μ ∗ (C, y 0 )=∞) if 

and only if there exists a function ψ(·) ∈ C 1 such that 

(3.4) max 

(y,u)∈Y ×U {∇ψ(y)T f(y, u)+C(ψ(y 0 ) − ψ(y))} < 0. 

Proof. The proof of the theorem is in the appendix. 

By taking C = 0 in (3.1), one obtains the problem 

{ 

(3.5) sup μ 

∣ ∃ψ(·) ∈ } 

C1 : ∀(y, u) ∈ Y × U, 

μ ≤∇ψ(y) T f(y, u)+g(y, u) 

def 

= μ ⋆ . 

As has been shown in [22] (see Theorem 4.1 in [22]), this problem is dual with respect 

to (1.7) in the sense that the dual relationships between the latter and (3.5) (similar 

to those established by Theorem 3.1) are true. The proof of Theorem 3.1 follows 

exactly the same steps as those of the above mentioned result of [22], and the latter 

can be viewed as a special case (C = 0) of Theorem 3.1. In fact, with C =0,the 

relationships (3.3) and (3.4) take the forms 

(3.6) μ ∗ def 

= μ ∗ (0,y 0 )=g ∗ (0,y 0 ) def 

= g ∗ , 

and, respectively, 

(3.7) max 

(y,u)∈Y ×U {∇ψ(y)T f(y, u)} < 0, 

with (3.6) being valid if and only if the set W (0,y 0 )=W is not empty and with (3.7) 

being satisfied for some ψ(·) ∈ C 1 if and only if this set is empty (the latter is also 

equivalent to μ ∗ = ∞). 

Note that duality results similar to Theorem 3.1(ii) have been obtained in [11] 

and [24] in a stochastic setting without state constraints (Y = R m ) and in [58] in the 

deterministic setting with state constraints (for IDLP problems related to optimal 

control problems considered on a finite time interval). Note also that problem (3.1) 

is equivalent to 



(3.8) sup 

ψ(·)∈C 1 

min 

(y,u)∈Y ×U {∇ψ(y)T f(y, u)+C(ψ(y 0 ) − ψ(y)) + g(y, u)} = μ ⋆ (C, y 0 ) 

and that max-min representations similar to (3.8) (with C = 0) have been studied for 

optimally controlled diffusions in [43] (see also [31] and references therein). 

Let us now extend the class of functions used in (3.1) from C 1 to Lip. 

Lemma 3.2. The following representation for μ ∗ (C, y 0 ) is valid: 

{ 

(3.9) μ ⋆ (C, y 0 )=sup μ 

∣ 

} 

∃ψ(·) ∈ Lip : ∀(y, u) ∈ Y × U, 

μ ≤ min ξ∈∂ψ(y) ξ T , 

f(y, u)+g(y, u)+C(ψ(y 0 ) − ψ(y)) 

where ∂ψ(y) stands for Clarke’s generalized gradient of ψ(y). 

Proof. Let us denote by ¯μ(C, y 0 ) the right-hand side of (3.9). It is easy to see 

that μ ∗ (C, y 0 ) ≤ ¯μ(C, y 0 ) and, hence, the statement is obvious if μ ∗ (C, y 0 )=+∞. 

Suppose that μ ∗ (C, y 0 ) < +∞, and take arbitrary μ ∈ R, ψ(·) ∈ Lip such that 

(3.10) μ ≤ min 

ξ∈∂ψ(y) ξT f(y, u)+g(y, u)+C(ψ(y 0 ) − ψ(y)) ∀(y, u) ∈ Y × U. 

Due to the fact that y ↦→ ∂φ(y) is upper semicontinuous and has convex and compact 

values, and due to the continuity of the functions f(·),g(·), ψ(˙), corresponding to any 

ε>0, there exists ν(ε) > 0 such that lim ε→0 ν(ε) =0andsuchthat 

(3.11) 

μ − ν(ε) ≤ min 

ξ∈∂ψ(y) ξT f(y, u)+g(y, u)+C(ψ(y 0 ) − ψ(y)) ∀(y, u) ∈ (Y + εB) × U, 

where B is the open unit ball with the center at the origin in R m . 

Fix ε ∈ (0, 1). By Theorem 2.2 in [19] (cf. also [53]), there exists ψ ε (·) ∈ C 1 such 

that, for any y ∈ Y + εB, 

(3.12) |ψ(y) − ψ ε (y)| ≤ε; 

(3.13) ∇ψ ε (y) ∈ 

⋃ 

y ′ ∈y+εB 

∂ψ(y ′ )+εB. 

def 

Let ‖f‖ ∞ =max (y,u)∈ Ŷ ×U 

||f(y, u)||, where||·|| stands for the Euclidean norm of 

a vector in R m and Ŷ is a compact set containing Y + εB for all ε small enough. Let 

L(f),L(g) denote Lipschitz constants in y of f(y, u),g(y, u), and let L(ψ) denotethe 

Lipschitz constant of ψ(y) fory from the set Ŷ . 

Fix arbitrary (y, u) ∈ Y × U. By (3.13), there exist y ε ∈ y + εB, ξ ε ∈ ∂ψ(y ε ), 

and b ε ∈ εB such that 

(3.14) ∇ψ ε (y) =ξ ε + b ε . 

Using this, one can obtain that 

∇ψ ε (y) T f(y, u)+g(y, u)+C(ψ ε (y 0 ) − ψ ε (y)) 

≥ ξ T ε f(y, u)+g(y, u)+C(ψ ε(y 0 ) − ψ ε (y)) − ε‖f‖ ∞ 

≥ ξ T ε f(y ε ,u)+g(y ε ,u)+C(ψ ε (y 0 ) − ψ ε (y)) − ε(‖f‖ ∞ + L(f)||ξ ε || + L(g)) 

≥ ξ T ε f(y ε,u)+g(y ε ,u)+C(ψ ε (y 0 ) − ψ ε (y)) − ε(‖f‖ ∞ + L(f)L(ψ)+L(g)) 



(since ξ ε ∈ ∂ψ(y ε ) ⊂ L(ψ) ¯B, with ¯B being the closure of B) 

≥ ξ T ε f(y ε,u)+g(y ε ,u)+C(ψ(y 0 ) − ψ(y)) − ε(‖f‖ ∞ + L(f)L(ψ)+L(g)+2C) 

(due to (3.12)) 

≥ ξ T ε f(y ε ,u)+g(y ε ,u)+C(ψ(y 0 ) − ψ(y ε )) − ε(‖f‖ ∞ + L(f)L(ψ)+L(g)+2C 

+ CεL(ψ)) 

≥ min ξ∈∂ψ(yε) ξ T f(y ε ,u)+g(y ε ,u)+C(ψ(y 0 ) − ψ(y ε )) − εR, 

where R def 

= ‖f‖ ∞ + L(f)L(ψ)+L(g)+2C + CL(ψ). By (3.11), this implies 

μ − εR − ν(ε) ≤∇ψ ε (y) T f(y, u)+g(y, u)+C(ψ ε (y 0 ) − ψ ε (y)). 

Since the latter is valid for any element (y, u) ∈ Y × U, itleadsto 

μ − εR − ν(ε) ≤ μ ∗ (C, y 0 ). 

This, in turn, leads to the inequality μ ≤ μ ∗ (C, y 0 )asε can be arbitrary small, and, 

consequently, to the inequality ¯μ(C, y 0 ) ≤ μ ∗ (C, y 0 )asμ, ψ(·) were chosen arbitrarily 

just to satisfy (3.10). 

The following lemma and its corollary establish that the set W (C, y 0 ) can also be 

characterized with the help of Lipschitz continuous functions. 

Lemma 3.3. If γ ∈ W (C, y 0 ), then, for any ψ(·) ∈ Lip, 

∫ ( 

) 

(3.15) 

min 

ξ∈∂ψ(y) ξT f(y, u)+C(ψ(y 0 ) − ψ(y)) γ(dy, du) ≤ 0. 

Y ×U 

Proof. Let ψ(·) ∈ Lip. As in the proof of Lemma 3.2, corresponding to any 

ε ∈ (0, 1), there exists a function ψ ε (·) ∈ C 1 such that (3.12) and (3.13) are valid, and 

for any (y, u) ∈ Y × U, 

∇ψ ε (y) T f(y, u)+C(ψ ε (y 0 ) − ψ ε (y)) ≥ 

min 

ξ∈∂ψ(y ξT f(y ε ,u)+C(ψ(y 0 ) − ψ(y ε )) − ε ˆR, 

ε) 

where y ε ∈ y + εB and ˆR >0 is a large enough constant. Due to the upper semicontinuity 

of the map y ↦→ ∂φ(y), due to the continuity of the functions f(·), ψ(·), 

and also due to the fact that the set Y is compact, there exists ˆν(ε) > 0 such that 

lim ε→0 ˆν(ε) =0andsuchthat 

min 

ξ∈∂ψ(y ξT f(y ε ,u) ≥ min 

ε) ξ∈∂ψ(y) ξT f(y, u) − ˆν(ε) 

⇒ ∇ψ ε (y) T f(y, u)+C(ψ ε (y 0 ) − ψ ε (y)) ≥ min 

ξ∈∂ψ(y) ξT f(y, u)+C(ψ(y 0 ) − ψ(y)) 

−ε ˆR − ˆν(ε). 

By integrating the last inequality over γ ∈ W (C, y 0 ), one obtains 

∫ ( 

) 

0 ≥ min 

ξ∈∂ψ(y) ξT f(y, u)+C(ψ(y 0 ) − ψ(y)) γ(dy, du) − ε ˆR − ˆν(ε). 

Y ×U 

Consequently, by letting ε → 0, one establishes (3.15). 



Corollary 3.4. The set W (C, y 0 ) allows the representations 

(3.16) { 

∫ ( 

) 

W (C, y 0 )= γ ∈P(Y ×U) : min 

ξ∈∂φ(y) ξT f(y, u)+C(φ(y 0 ) − φ(y)) γ(dy, du) 

Y ×U 

≤ 0 

} 

∀φ ∈ Lip . 

Proof. By Lemma 3.3, W (C, y 0 ) is contained in the set defined by the right-hand 

side of (3.16). To prove the opposite inclusion, take an arbitrary γ belonging to the 

latter set. Then, for any φ(·) ∈ C 1 , 

∫ 

(∇φ(y) T f(y, u)+C(φ(y 0 ) − φ(y)))γ(dy, du) ≤ 0, 

∫ 

Y ×U 

Y ×U 

(∇(−φ(y)) T f(y, u)+C((−φ(y 0 )) − (−φ(y))))γ(dy, du) ≤ 0 

⇒ 

∫ 

Y ×U 

(∇φ(y) T f(y, u)+C(φ(y 0 ) − φ(y)))γ(dy, du) =0. 

This implies that γ ∈ W (C, y 0 ). 

Finally, let us establish necessary and sufficient optimality conditions for γ ∗ ∈ 

W (C, y 0 ) to be optimal in (1.4). 

Lemma 3.5. Let γ ∗ ∈ W (C, y 0 ),andletthereexistψ(·) such that 

(3.17) μ ⋆ (C, y 0 ) ≤ min 

ξ∈∂ψ(y) ξT f(y, u)+g(y, u)+C(ψ(y 0 ) − ψ(y)) ∀(y, u) ∈ Y × U. 

Then, for γ ∗ to be an optimal solution of the problem (1.4) it is necessary and sufficient 

that the following relationships are satisfied: 

∫ ( 

) 

(3.18) 

min 

ξ∈∂ψ(y) ξT f(y, u)+C(ψ(y 0 ) − ψ(y)) γ ∗ (dy, du) =0; 

Y ×U 

(3.19) γ ∗ (Ω(C, y 0 )) = 1, 

where 

(3.20) Ω(C, y 0 ) def 

= 

{ 

(y, u) ∈ Y × U : 

min 

ξ∈∂ψ(y) ξT f(y, u)+g(y, u) 

} 

+ C(ψ(y 0 ) − ψ(y)) = μ ⋆ (C, y 0 ) . 

Proof. Let γ ∗ ∈ W (C, y 0 ) be an optimal solution of (1.4). That is, 

∫ 

g(y, u)γ ∗ (dy, du) =g ∗ (C, y 0 ). 

Y ×U 



By integrating (3.17) and taking into account (3.3), one can obtain that 

∫ ( 

) 

min 

ξ∈∂ψ(y) ξT f(y, u)+C(ψ(y 0 ) − ψ(y)) γ ∗ (dy, du) ≥ 0. 

Y ×U 

The latter together with Lemma 3.3 imply (3.18). Assume that (3.19) is not true. 

Then, by integrating (3.17) and taking into account (3.18), one would obtain that 

μ ∗ (C, y 0 )


4. Optimal control problem (1.3) and IDLP problem (1.4). In this section 

we use results of previous sections to show that, under mild conditions, inequality 

(2.12) and inclusion (2.13) relating optimal control problem (1.3) and its LP counterpart 

(1.4) take the form of equalities. We also use necessary and sufficient optimality 

conditions for (1.4) to construct necessary and sufficient optimality conditions for 

(1.3). 

Along with problem (1.3) let us consider the problem 

(4.1) inf 

u(·)∈U 

∫ +∞ 

0 

e −Ct g(y(t, y 0 ,u(·)),u(t))dt def 

= V C (y 0 ) 

and also a family of optimal control problems parameterized by δ (δ >0), 

(4.2) inf 

u(·)∈U Y δ (y 0) 

∫ +∞ 

0 

e −Ct g(y(t, y 0 ,u(·))u(t))dt def 

= V Y δ 

C (y 0), 

where U Y δ 

(y 0 ) ⊂ U is the set of controls such that, for any u(·) ∈ U Y δ 

(y 0 ), the 

associated trajectory satisfies the inclusion 

y(t, y 0 ,u(·)) ∈ Y + δ 

¯B 

def 

= Y δ ∀t ≥ 0, 

with ¯B a closed unit ball of R m as above. Note that, for any δ>0, 

(4.3) V C (y 0 ) ≤ V Y δ 

C (y 0) ≤ V Y C (y 0 ) ∀y 0 ∈ Y. 

V Y δ 

C 

Lemma 4.1. Let U Y δ 

(y 0 ) be not empty for any y 0 ∈ Y δ , and let the function 

(·) (δ being fixed) satisfy Lipschitz conditions on Y δ.Then 

(4.4) CV Y δ 

C (y 0) ≤ μ ∗ (C, y 0 ) ∀y 0 ∈ Y. 

Proof. Let 

(4.5) H(y, ξ) def 

=max 

u∈U {−ξT f(y, u) − g(y, u)}. 

It is known that the function V Y δ 

C 

(·) is a viscosity solution of the Hamilton–Jacobi– 

Bellman (HJB) equation 

(4.6) CV Y δ 

C 

+ H(y, DV Y δ 

C )=0, 

in intY δ (the interior of Y δ ). See Proposition III.2.8 on page 104 and comments on page 

277 in [7]. Being a viscosity solution (and due to the fact that it satisfies Lipschitz 

conditions on Y δ ), the function V Y δ 

C 

(·) also solves the HJB in the extended sense in 

intY δ (see Proposition II.5.13 on page 85 in [7] and also see results in [25]; further 

information on the topic can be found in [16] and [18]). Namely, 

(4.7) CV Y δ 

C 

(y)+ max 

ξ∈∂V Y δ 

C (y) H(y, ξ) =0 ∀y ∈ intY δ . 

As Y ⊂ intY δ , from (4.5) and (4.7) it follows that 



{ 

} 

(4.8) −CV Y δ 

C 

(y)+min min ξ T f(y, u)+g(y, u) =0 ∀y ∈ Y 

u∈U ξ∈∂V Y δ 

C (y) 

(4.9) ⇒ −CV Y δ 

C 

(y)+ min 


C (y) ξ T f(y, u)+g(y, u) ≥ 0 ∀(y, u) ∈ Y × U 

⇒ C(V Y δ 

C 

(y 0)−V Y δ 

C 

(y))+ min 


C 

(y) ξ T f(y, u)+g(y, u) ≥ CV Y δ 

C (y 0) ∀(y, u) ∈ Y ×U. 

By (3.9) in Lemma 3.2, the latter implies (4.4). 

Corollary 4.2. If V C (·) satisfies Lipschitz conditions in a neighborhood of Y , 

then 

(4.10) CV C (y 0 ) ≤ μ ∗ (C, y 0 ) ∀y 0 ∈ Y ; 

(4.11) min 

y∈Y CV C(y) ≤ μ ∗ . 

Proof. Similarly to the proof above (see (4.9)) it is established that 

(4.12) −CV C (y)+ min 

ξ∈∂V C(y) ξT f(y, u)+g(y, u) ≥ 0 

∀(y, u) ∈ Y × U 

⇒ C(V C (y 0 ) − V C (y)) + min 

ξ∈∂V ξT f(y, u)+g(y, u) ≥ CV C (y 0 ) ∀(y, u) ∈ Y × U. 

C(y) 

This implies the validity of (4.10). From (4.12) it also follows that 

min 

ξ∈∂V C (y) ξT f(y, u)+g(y, u) ≥ min 

y ′ ∈Y CV C(y ′ ) ∀(y, u) ∈ Y × U, 

which, by (3.21) in Remark 3.6, implies (4.11). 

Let us now introduce an assumption that would ensure that 

(4.13) lim V Y δ 

C (y 0)=VC Y (y 0 ) ∀y 0 ∈ Y 

δ→0 

and, thus, by passing to the limit with δ → 0 in (4.4), would lead to the inequality 

(4.14) CV Y C (y 0) ≤ μ ∗ (C, y 0 ) ∀y 0 ∈ Y. 

Note that (4.13) is satisfied automatically if Y is invariant with respect to the solutions 

of system (1.1), in which case, for any δ>0, 

(4.15) V C (y 0 )=V Y δ 

C (y 0)=VC Y (y 0 ) ∀y 0 ∈ Y. 

Let us denote by P(U) the space of probability measures defined on the Borel 

subsets of U, and let us consider the so-called relaxed control system (see [59]) 

(4.16) ẏ(t) = ¯f(y(t),v(t)), t ≥ 0, 

where 

(4.17) ¯f(y, v) 

def 

= 

∫ 

U 

f(y, u)v(du) 



and the controls (“relaxed controls”) are Lebesgue measurable functions v(·) :[0+ 

∞) ↦→ P(U). Given a relaxed control v(·), let us denote by t ↦→ y(t, y 0 ,v(·)) the 

solution of system (4.16) obtained with this control and with the initial condition 

y(0) = y 0 . Let V Y (y 0 ) stand for the set of relaxed controls such that, for any v(·) ∈ 

V Y (y 0 ), the associated trajectory satisfies the inclusion y(t, y 0 ,v(·)) ∈ Y for all t ≥ 0. 

Note that U Y (y 0 ) can be considered a subset of V Y (y 0 )thatconsistsofonlyDirac 

measure-valued functions. 

Assumption I. For any q(·) ∈ C(Y × U), 

inf 

u(·)∈U Y (y 0) 

∫ +∞ 

0 

e −Ct q(y(t, y 0 ,u(·)),u(t))dt 

= inf 

v(·)∈V Y (y 0) 

∫ +∞ 

0 

e −Ct¯q(y(t, y 0 ,v(·)),v(t))dt ∀y 0 ∈ Y, 

where ¯q(y, v) def 

= ∫ q(y, u)v(du). 

U 

Sufficient conditions for Assumption I to be satisfied are those that ensure the 

applicability of Filippov–Wazewski type theorems on Y (see [26]). In particular, 

Assumption I is satisfied if Y is invariant with respect to the solutions of system 

(1.1) or if f(y, u) ≡ f(y) (the case of uncontrolled dynamics). Some other types 

of conditions, under which Assumption I is satisfied, have been described in [29]. 

For example, it is satisfied if U and Y are convex, f(y, u) is linear, and there exist 

ȳ ∈ intY, ū ∈ U such that f(ȳ, ū) = 0 (see Proposition 4.3 in [29]). 

Assumption I is not satisfied if, for example, U Y (y 0 ) = ∅, while V Y δ 

(y 0 ) ≠ 

∅ for all y 0 ∈ Y ,asisthecasewhen m =1, f(y, u) =−y + u, with U consisting 

of two points, U = {−1, 1}, and Y consisting of one point, Y = {0}. See 

Assumption 1 and Remark 1 in [30]. 

Lemma 4.3. Let U Y (y 0 ) ≠ ∅ for all y o ∈ Y , and let Assumption I be satisfied. 

Then (4.13) is valid. 

Proof. The proof follows a standard argument. An outline of the proof is given 

in the appendix. 

Theorem 4.4. Let U Y (y 0 ) ≠ ∅ for all y 0 ∈ Y and let U Y δ 

(y 0 ) ≠ ∅ for all y 0 ∈ Y δ 

for any δ ∈ (0,δ 0 ] (δ 0 > 0 being small enough). Let also Assumption I be satisfied 

and the function V Y δ 

C 

(·) satisfy Lipschitz conditions on Y δ.Then 

(4.18) CV Y C (y 0 )=g ∗ (C, y 0 ) ∀y 0 ∈ Y. 

If, for any Lipschitz continuous q(·) :R m × U → R, the function V Y δ 

C,q (·), 

(4.19) V Y δ 

C,q (y 0) def 

= inf 

u(·)∈U Y δ (y 0) 

∫ +∞ 

satisfies Lipschitz conditions on Y δ ,then 

(4.20) ¯coΓ(C, y 0 )=W (C, y 0 ). 

0 

e −Ct q(y(t, y 0 ,u(·)),u(t))dt, 

Proof. By Lemma 4.3, one can pass to the limit with δ → 0 in (4.4) to obtain 

(4.14). The latter and (2.12) imply (4.18). From the fact that V Y δ 

C,q 

(·) satisfies Lipschitz 

conditions on Y δ (with the other conditions of the theorem assumed to be satisfied) it 

follows that the equality similar to (4.18) is valid with the replacement of g(·) byany 



Lipschitz continuous q(·). In the notation introduced in section 2, this can be written 

as 

∫ 

∫ 

(4.21) inf q(y, u)γ(dy, du) = min q(y, u)γ(dy, du). 

γ∈Γ(C,y 0) Y ×U 

γ∈W (C,y 0) Y ×U 

Since 

∫ 

∫ 

inf q(y, u)γ(dy, du) = min q(y, u)γ(dy, du), 

γ∈Γ(C,y 0) Y ×U 

γ∈ ¯coΓ(C,y 0) Y ×U 

from (4.21) it follows that 

∫ 

(4.22) min 

γ∈ ¯coΓ(C,y 0) 

Y ×U 

q(y, u)γ(dy, du) = 

∫ 

min q(y, u)γ(dy, du). 

γ∈W (C,y 0) Y ×U 

Due to the fact that the space of Lipschitz continuous functions is dense in C(Y × U), 

the latter will be also valid for any q(·) ∈ C(Y × U). This implies (4.20). 

Remark 4.5. Note that the introduction of Assumption I would not be necessary 

(it would just be satisfied automatically) if the problem (1.4) was formulated in the 

relaxed control setting (that is, if, instead of (1.1), we used system (4.16) with controls 

v(·) ∈V Y (y 0 )). Note also that the assumption about Lipschitz continuity of V Y δ 

C (·) 

used in Theorem 4.4 is a technical one (that is, it is related to the argument used in the 

proof of the theorem). In fact, relationships similar to (4.18) and (4.20) can be proved 

without this assumption (e.g., by modifying the approach of [58] to make it applicable 

to problems considered on the infinite time horizon). Such a proof, however, is much 

more involved, and, hence, is not included in this paper (to keep the presentation 

expository). 

To conclude this section, let us construct necessary and sufficient optimality conditions 

for the optimal control problem (1.3) based on the assumption that a function 

ψ(·) satisfying (3.17) exists. For every y ∈ Y ,let 

{ 

} 

K(y) def 

=min 

u∈U 

min 

ξ∈∂ψ(y) ξT f(y, u)+g(y, u) 

{ 

} 

(4.23) D(y) def 

= u ∈ U : K(y) = min 

ξ∈∂ψ(y) ξT f(y, u)+g(y, u) , 

and let 

(4.24) Y def 

= {y ∈ Y : K(y)+C(ψ(y 0 ) − ψ(y)) = μ ⋆ (C, y 0 )}. 

Note that, in this notation, the set Ω(C, y 0 ) introduced in (3.20) is presented in the 

form 

(4.25) Ω(C, y 0 )={(y, u) ∈ Y × U : u ∈D(y), y ∈Y}. 

Proposition 4.6. Let a function ψ(·) satisfying (3.17) exist. Then for a control 

u ∗ (·) ∈U Y (y 0 ) to be optimal in (1.4) and for the equality (4.18) to be true, it is 

necessary and sufficient that 

∫ ∞ 

( 

) 

(4.26) e −Ct min 

ξ∈∂ψ(y ∗ (t)) ξT f(y ∗ (t),u ∗ (t)) + C(ψ(y(t)) − ψ(y 0 )) dt =0 

0 

, 



and 

(4.27) u ∗ (t) ∈D(y ∗ (t)), y ∗ (t) ∈Y, 

for almost all t ∈ [0, ∞), wherey ∗ (t) def 

= y(t, y0 ∗,u∗ (·)). 

Proof. Let u ∗ (·) ∈U Y (y 0 ) be optimal in (1.4) and let (4.18) be satisfied. Then 

the discounted occupational measure γu C def 

∗ (·) 

= γ ∗ generated by the pair (y ∗ (·),u ∗ (·)) 

on the interval [0, ∞) is a solution of IDLP problem (1.4). Hence, by Lemma 3.5, the 

relationships (3.18) and (3.19) are satisfied, (3.18) is equivalent to (4.26), and (3.19) 

is equivalent to the fact that the inclusion (y ∗ (t),u ∗ (t)) ∈ Ω(C, y 0 ) is valid for almost 

all t ∈ [0, ∞). The validity of the latter is equivalent to (4.27) (due to (4.25)). 

Conversely, let (4.26) and (4.27) be satisfied. Then the discounted occupational 

measure γ ∗ generated by the pair (y ∗ (·),u ∗ (·)) satisfies (3.18) and (3.19). 

By ∫ Lemma 3.5, it implies that γ ∗ is a solution of IDLP problem (1.4). Hence, 

∞ 

e −Ct g(y(t),u(t))dt = g ∗ (C, y 

0 0 ). Thisprovesthatu ∗ (·) ∈ U Y (y 0 )isoptimal 

and that (4.18) is satisfied. 

Note that in a special case when Y is invariant, equalities (4.15) are valid, and 

from (4.8) it follows that 

(4.28) −CV C (y)+min 

u∈U 

{ 

min 

ξ∈∂V C (y) ξT f(y, u)+g(y, u) 

} 

=0 ∀y ∈ Y, 

which implies that (3.17) is satisfied with ψ(y) =V (C, y). 

5. Characterization of the viability kernel of Y . In this section we demonstrate 

the possibility of applying one of the results obtained above (namely, Corollary 

4.2 of section 4) for a characterization of the viability kernel of Y . 

Let us note, first, that the viability kernel of Y (denoted as Viab f(·,U) ; see [5], 

[48], [49]) is defined as a “largest” subset of Y such that, for any point y 0 belonging 

to this subset, U Y (y 0 ) ≠ ∅. Thatis, 

y 0 ∈ Viab f(·,U) ⇔ U Y (y 0 ) ≠ ∅. 

We use here an idea of [49] for the characterization of the viability of Y . Todothis, 

consider problem (4.1) with 

(5.1) g(y, u) =d 2 Y (y), d Y (y) def 

= min 

y ′ ∈Y ||y − y′ ||. 

Proposition 5.1. Let the set f(y, U) def 

= {η : η = f(y, u), u ∈ U} be convex for 

any y ∈ Y .Theny 0 ∈ Viab f(·,U) if and only if W (C, y 0 ) ≠ ∅. 

Proof. If y 0 ∈ Viab f(·,U) (Y ), then Γ(C, y 0 ) ≠ ∅ and, consequently, W (C, y 0 ) ≠ ∅ 

(see Proposition 2.2). Conversely, if W (C, y 0 ) ≠ ∅, then ∫ Y ×U d2 Y (y)γ(dy, du) =0for 

any γ ∈ W (C, y 0 ), and 

g ∗ (C, y 0 )=μ ∗ (C, y 0 )=0 

if g(·) in problems (1.4) and (3.1) is defined by (5.1). By choosing C large enough, 

one may assume (without loss of generality) that the function V C (·) satisfies Lipschitz 

conditions on R m . See, e.g., Proposition III.2.1 on page 99 in [7]. Hence, by Corollary 

4.2 (see 4.10), CV C (y 0 ) ≤ 0, which implies that V C (y 0 ) = 0 (since the value function 

is not negative with g(·) as in (5.1)). Consequently, 

(5.2) inf 

u(·)∈U 

∫ +∞ 

0 

e −Ct d 2 Y (y(t, y 0,u(·)))dt =0. 



Due to the fact that y ↦→ f(y, U) is compact and convex valued, the optimal control 

in the above problem exists. The corresponding solution of (1.1) is contained in Y . 

Hence, U Y (y 0 ) ≠ ∅. 

Proposition 5.2. Let f(y, U) be convex for any y ∈ Y . Then Viab f(·,U) ≠ ∅ if 

and only if W ≠ ∅. 

Proof. If Viab f(·,U) (Y ) ≠ ∅, thenΓ S ≠ ∅ for all S>0, and, consequently, W ≠ ∅ 

(see Proposition 2.3). Conversely, let W ≠ ∅. Then ∫ Y ×U d2 Y (y)γ(dy, du) = 0 for any 

γ ∈ W ,and 

g ∗ = μ ∗ =0 

if g(·) in problems (1.7) and (3.5) is defined by (5.1). For C large enough, the optimal 

value function V C (·) of problem (4.1) satisfies Lipschitz conditions on R m , and, hence, 

one can use (4.11) to obtain 

C min 

y∈Y V C(y) ≤ 0 ⇒ min 

y∈Y V C(y) =0 

(the last equality being implied by the fact that V C (·) is nonnegative). It follows that 

there exists y 0 ∈ Y such that V C (y 0 ) = 0. That is, (5.2) is valid. Repeating now 

the argument used in the proof of Proposition 5.1, one arrives at the conclusion that 

Viab f(·,U) (Y ) ≠ ∅. 

Remark 5.3. Note that, in the relaxed control setting, Propositions 5.1 and 5.2 

are valid without the assumption about the convexity of f(y, U) (see[49]andRemark 

4.5 above). 

Remark 5.4. From Proposition 5.1 and Theorem 3.1 it follows that y 0 ∈ Viab f(·,U) 

if and only if there does not exist a function ψ(·) satisfying (3.4). Also from Proposition 

5.2 and Theorem 3.1 (considered with C = 0), it follows that Viab f(·,U) ≠ ∅ if 

and only if there does not exist a function ψ(·) satisfying (3.7). The latter function 

can be interpreted as a Lyapunov function, its existence forcing the solutions of (1.1) 

to leave Y in a finite time (see Theorem 4.1 and Remark 4.2 in [22]). 

6. Optimal control problems with long run average criteria. In this section 

we show that one can use Theorem 4.4 of section 4 and Abelian-type results 

obtained in [32], [33] to establish relationships between the optimal control problem 

with long run average criteria (that is, problem (1.6) with S →∞)andtheIDLP 

problem (1.7). Note that similar relationships have been established in Theorem 

2.1(i) of [28] and in Proposition 5 of [30] under weaker assumptions than those used 

in this section. However, the considerations in [28] and [30] were based on results from 

stochastic control theory (see [11], [39], and [54]), the proof of which being rather sophisticated. 

The proof presented below is straightforward and is based on a purely 

deterministic argument. 

Proposition 6.1. Let (i) U Y (y 0 ) ≠ ∅ for all y 0 ∈ Y and U Y δ 

(y 0 ) ≠ ∅ for all y 0 ∈ 

Y δ for any δ ∈ (0,δ 0 ] (δ 0 > 0 being small enough); (ii) Assumption I be valid for every 

C ∈ (0,C 0 ] (C 0 being a given positive number); and (iii) the function V Y δ 

C 

(·) satisfy 

Lipschitz conditions on Y δ (with a Lipschitz constant being independent on δ ∈ (0,δ 0 ]) 

for every C ∈ (0,C 0 ]. Then there exists the limit 

(6.1) lim 

C→0 

min 

y∈Y CV Y C (y) =g∗ . 



If also (iv) for every C ∈ (0,C 0 ] and any Lipschitz continuous q(·) :R m × U → R, 

the function V Y δ 

C,q (·) defined by (4.19) satisfies Lipschitz conditions on Y δ,then 

(6.2) lim ρ H (¯ coΓ(C),W)=0, 

C→0 

Proof. By (4.9), 

min 


C 

Hence, by (3.21) and (3.3), 

Γ(C) def 

= ⋃ 

y 0∈Y 

Γ(C, y 0 ). 

(y) 

ξ T f(y, u)+g(y, u) ≥ min 

y ′ ∈Y CV Y δ 

C (y′ ) ∀(y, u) ∈ Y × U. 

(6.3) g ∗ ≥ min 

y∈Y CV Y δ 

C (y) ⇒ g∗ ≥ min 

y∈Y CV Y C (y), 

where the latter is obtained via passing to the limit in the former (and taking into 

account (4.13) as well as the fact that V Y δ 

C 

(·) satisfies Lipschitz conditions with a 

constant independent of δ). Since, by (2.19) and (2.12), 

(6.4) lim C→0 min 

y∈Y CV Y C (y) ≥ g∗ , 

the validity of (6.1) is established. 

Under the additional assumption that the function V Y δ 

C,q 

(·) defined by (4.19) satisfies 

Lipschitz conditions on Y δ for every C ∈ (0,C 0 ] and for any Lipschitz continuous 

q(·) :R m × U → R, the equality similar to (6.1) is valid with the replacement of g(·) 

by any Lipschitz continuous q(·). Due to the definition of Γ(C) (see (6.2)), this can 

be written as 

∫ 

∫ 

lim inf q(y, u)γ(dy, du) =min q(y, u)γ(dy, du) 

C→0 γ∈Γ(C) Y ×U 

γ∈W Y ×U 

(6.5) ⇒ lim 

min 

C→0 γ∈ ¯coΓ(C) 

∫ 

Y ×U 

∫ 

q(y, u)γ(dy, du) =min q(y, u)γ(dy, du). 

γ∈W Y ×U 

Since the space of Lipschitz continuous functions is dense in C(Y × U), the latter is 

also valid for any q(·) ∈ C(Y × U). Let us now use this fact to prove (6.2). 

First, note that, by (2.21) and (2.13), 

lim max ρ(γ,W)=0. 

C→0 γ∈ ¯coΓ(C) 

Thus, to prove (6.2), one needs to show that 

(6.6) lim 

max 

C→0 γ∈W 

ρ(γ, ¯coΓ(C)) = 0. 

Assume it is not true. Then there exists a positive number α and sequences C i > 0, 

γ i ∈ W , i =1, 2,..., such that lim i→∞ C i =0 and 

(6.7) lim ρ(γ i , coΓ(C ¯ i )) ≥ α. 

i→∞ 

Due to the fact that W is compact in metric ρ, and due to the fact that (by Blaschke’s 

selection theorem; see, e.g., [38]) the set of closed subsets of P(Y × U) iscompactin 



metric ρ H , the sequences {γ i } and { ¯coΓ(C i )} have partial limits. More specifically, 

there exist ˆγ ∈ W and ˆΓ ⊂P(Y × U) such that, for some subsequence {i ′ }⊂{i}, 

(6.8) lim ρ(γ i ′, ˆγ) =0, lim 

i ′ →∞ 

ρ H(¯ coΓ(C i ′), ˆΓ) = 0. 

i ′ →∞ 

Then, by passing to the limit in (6.7), one obtains that 

(6.9) ρ(ˆγ, ˆΓ) ≥ α ⇒ ˆγ /∈ ˆΓ. 

By the separation theorem (see, e.g., [52, p. 59]), there exists ˆq(·) ∈ C(Y × U) such 

that 

∫ 

∫ 

ˆq(y, u)ˆγ(dy, du) ≤ min ˆq(y, u)γ(dy, du) − β, 

Y ×U 

γ∈ˆΓ Y ×U 

where β>0 is some constant. Hence, 

∫ 

(6.10) min 

γ∈W Y ×U 

From (6.8) it follows that 

lim 

min 

i ′ →∞ γ∈ ¯coΓ(C i ′ ) 

∫ 

ˆq(y, u)γ(dy, du) ≤ min ˆq(y, u)γ(dy, du) − β. 


∫ 

Y ×U 

∫ 

ˆq(y, u)γ(dy, du) =min ˆq(y, u)γ(dy, du). 


Consequently, by (6.10), 

∫ 

∫ 

(6.11) min ˆq(y, u)γ(dy, du) ≤ min ˆq(y, u)γ(dy, du) − β 

γ∈W Y ×U 

γ∈ ¯coΓ(C i ′ ) Y ×U 

2 

for i ′ large enough. This contradicts the fact that (6.5) is valid with any q(·) ∈ C(Y × 

U). The contradiction proves (6.6) and, thus, it completes the proof of (6.2). 

Proposition 6.2. Let conditions (i), (ii), and(iii) of Proposition 6.1 be satisfied. 

Then 

(6.12) lim 

S→∞ G S = g ∗ . 

If also condition (iv) of Proposition 6.1 is satisfied, then 

(6.13) lim ρ H(¯ coΓ S ,W)=0. 

S→∞ 

Proof. Let C i → 0. By (6.1), there exists a sequence of controls u i (·) ∈US Y (yi 0) 

and the corresponding sequence of solutions y i (t) def 

= y(t, y0 i ,ui (·)) of system (1.1) such 

that 

(6.14) lim 

i→∞ 

ζ i =0, 

def 

ζ i = 

∫ +∞ 

0 

e −Cit g(y i (t),u i (t))dt − g ∗ . 

From Lemma 3.5(ii) in [33] it follows that there exists a sequence S i , i =1, 2,...,such 

that S i ≥ (K>0 being a constant) and such that 

K √ Ci 



(6.15) 

∫ 

1 

Si 

g(y i (t),u i (t))dt ≤ g ∗ + ζ i + √ C i 

S i 0 

⇒ G Si ≤ g ∗ + ζ i + √ C i ⇒ lim S→∞ G S ≤ g ∗ . 

The latter inequality and (2.14) imply that 

(6.16) lim S→∞ G S = g ∗ , 

which, by (6.15), implies that 

(6.17) lim 

i→∞ 

η i =0, 

def 

η i = 1 ∫ Si 

S i 

0 

g(y i (t),u i (t))dt − g ∗ . 

From Lemma 3.8 in [32] it follows that, for any i, there exists a nonnegative t i ≤ 

S i − √ S i 

L 

(L>0 being a constant) such that 

∫ 

1 S 

(6.18) 

g(y i (t i + t),u i (t i + t))dt ≤ g ∗ + η i + 1 ( √ ] 

Si 

√ ∀S ∈ 0, . 

S 0 

Si 

L 

Let ũ i (·) def 

= u i (t i + ·), ỹ i (·) def 

= y i (t i + ·). Note that ũ i (·) ∈U Y (y i (t i )) and ỹ i (t) = 

y(t, y i (t i ), ũ i (·)). Hence, by (6.18), 

G S ≤ 1 S 

∫ S 

0 

g(ỹ i (t), ỹ i (t))dt ≤ g ∗ + η i + 1 √ 

Si 

∀S ∈ 

( 

0, 

√ 

Si 

L 

] 

⇒ lim S→∞ G S ≤ g ∗ . 

The latter and (6.16) prove (6.12). 

If condition (iv) of Proposition 6.1 is satisfied, then the equality similar to (6.12) 

is valid with the replacement of g(·) by any Lipschitz continuous q(·). In the notation 

of section 2, this can be written as 

∫ 

lim inf q(y, u)γ(dy, du) =min q(y, u)γ(dy, du) 

S→∞ 

γ∈W 

(6.19) ⇒ lim 

γ∈Γ S 

∫Y ×U 

min 

S→∞ γ∈ ¯coΓ S 

∫Y ×U 

Y ×U 

∫ 

q(y, u)γ(dy, du) =min q(y, u)γ(dy, du). 

γ∈W Y ×U 

Since the space of Lipschitz continuous functions is dense in C(Y × U), the latter is 

valid for any q(·) ∈ C(Y × U). From this point, the proof of (6.13) follows exactly the 

same lines as that of (6.2) in Proposition 6.1. 

Let us consider two special cases, in which the conditions of the above results are 

readily verifiable. 

Special case 1. Let there exist positive definite matrices A 1 and A 2 such that 

(6.20) (f(y ′ ,u) − f(y ′′ ,u)) T A 1 (y ′ − y ′′ ) 

≤−(y ′ − y ′′ ) T A 2 (y ′ − y ′′ ) ∀y ′ ,y ′′ ∈ R m , ∀u ∈ U. 



Then, for any u(·) ∈U, the solutions of system (1.1) satisfy the inequality 

(6.21) d 

dt (y(t, y′ 0 ,u(·)) − y(t, y′′ 0 ,u(·)))T A 1 (y(t, y 0 ′ ,u(·)) − y(t, y′′ 0 ,u(·))) 

≤−2(y(t, y 0 ′ ,u(·)) − y(t, y′′ 0 ,u(·)))T A 2 (y(t, y 0 ′ ,u(·)) − y(t, y′′ 0 ,u(·))) 

≤−α 1 (y(t, y 0,u(·)) ′ − y(t, y 0 ′′ ,u(·))) T A 1 (y(t, y 0,u(·)) ′ − y(t, y 0 ′′ ,u(·))) 

⇒ ||y(t, y 0 ′ ,u(·)) − y(t, y′′ 0 ,u(·))|| ≤ α 2||y 0 ′ − y′′ 0 ||e−α3t , 

where α i > 0,i =1, 2, 3, are appropriate constants. Hence, for any q(y, u) satisfying 

Lipschitz conditions in y with a constant L q , 

∫ ∞ 

∫ ∞ 

∥ ∥∥∥ 

∥ e −Ct q(y(t, y 0 ′ ,u(·)),u(t))dt − e −Ct q(y(t, y 0 ′′ ,u(·)),u(t))dt 

0 

0 

≤ 

∫ ∞ 

0 

e −Ct L q ||y(t, y ′ 0,u(·)) − y(t, y ′′ 

0 ,u(·))||dt ≤ ˆL q ||y ′ 0 − y ′′ 

0 ||, 

where ˆL 

def 

q = α2Lq 

α 3 

> α2Lq 

(6.22) V C,q (y 0 ) def 

= inf 

u(·)∈U 

C+α 3 

. It follows that the optimal value function V C,q (·), 

∫ +∞ 

0 

e −Ct q(y(t, y 0 ,u(·)),u(t))dt, 

satisfies Lipschitz conditions on R m with a constant ˆL q . Also, from the validity of 

the estimate (6.22) it follows that system (1.1) has an invariant set, which is a global 

attractor to all its solutions (see Theorem 3.1(ii) in [27]). Taking this set as Y ,one 

can arrive at the conclusion that the statements of Theorem 4.4 and Propositions 

6.1 and 6.2 are valid. Note that (6.20) is a Liapunov-type stability condition. It is 

satisfied, for example, if f(y, u) =Ay + f 1 (u), where A is a “stable” matrix (that is, 

its eigenvalues have negative real parts) or if, for all y and u, the eigenvalues of the 

) T are less than some negative constant. 

Special case 2. Assume that the set Y is convex and that there exist δ 0 > 0and 

r>0 such that 

matrix ∂f(y,u) 

∂y 

+( ∂f(y,u) 

∂y 

(6.23) y + rB ∈ f(y, U) ∀y ∈ Y δ0 . 

This is a controllability-type condition implying that any two points y 0 ′ and y 0 ′′ in Y δ 

and in Y are connectable by a trajectory of (1.1) lying in y δ (respectively, in Y ), with 

the time required for the transition from one point to another being equal to ‖y′ 0 −y′′ 0 ‖ 

r 

. 

It can be verified that, under this condition, the value function V Y δ 

C,q 

(·) satisfies Lipschitz 

conditions on Y δ with a constant ˆL q = Lq 

r ,whereL def 

q =max (y,u)∈Yδ0 ×U |q(y, u)| 

(see, e.g., page 398 in [7]). If, in addition, Assumption I is satisfied, then again the 

statements of Theorem 4.4 and Propositions 6.1 and 6.2 are valid. Note that in the 

relaxed control setting, the condition (6.23) takes the form y + rB ∈ ¯cof(y, U) for all 

y ∈ Y δ0 . 

7. Finite dimensional approximations. Numerical example. In this section 

we show that problems (1.4) and (3.1) can be written in a “standard” LP form, 

and we discuss the possibility of approximating these problems with finite dimensional 

LP problems. We also illustrate that the latter can be used for finding a near optimal 

control in (1.3) with a numerical example. 



Let {φ i (·) ∈ C 1 ,i=1, 2,...} be a sequence of functions having continuous partial 

derivatives of the second order such that any function ψ(·) ∈ C 1 and its gradient 

∇ψ(·) can be simultaneously approximated on Y by linear combinations of functions 

from {φ i (·),i=1, 2,...} and their corresponding gradients. That is, for any ψ(·) ∈ C 1 

and any δ>0, there exist β 1 ,...,β k (real numbers) such that 

{∣ ∥ } 

∣∣∣∣ k∑ 

∥∥∥∥ k∑ 

(7.1) max ψ(y) − β i φ i (y) 

y∈Y 

∣ + ∇ψ(y) − β i ∇φ i (y) 

≤ δ, 

∥ 

1 

with || · || being a norm in R m . An example of such an approximating sequence is the 

sequence of monomials y i1 

1 ...yim m , i 1,...,i m =0, 1,...,wherey j (j =1,...,m) stands 

for the jth component of y (see, e.g., [45]). 

Due to the above property of the sequence of the functions φ i (·),i=1, 2,..., the 

set W (C, y 0 ) can be presented in the form 

(7.2) { 

∫ 

W (C, y 0 ) def 

= γ ∈P(Y × U) : (∇φ i (y) T f(y, u)+C(φ i (y 0 ) − φ i (y)))γ(dy, du) 

Y ×U 

1 

} 

=0 i =1, 2,... , 

where, without loss of generality, one may assume that the functions φ i (·) satisfythe 

following normalization conditions: 

(7.3) max y∈ ˆD{|φ i (y)|, ||∇φ i (y)||} ≤ 1 , i =1, 2,..., 

2i where ||∇φ i (y)|| is a norm of ∇φ i (y) inR m ,and ˆD is a closed ball in R m that contains 

Y in its interior. 

Let l 1 and l ∞ stand for the Banach spaces of infinite sequences such that, for 

def 

any x =(x 1 ,x 2 ,...) ∈ l 1 , ||x|| l1 = ∑ i |x i| < ∞ and, for any λ =(λ 1 ,λ 2 ,...) ∈ l ∞ , 

def 

||λ|| l∞ =sup i |λ i | < ∞. It is easy to see that, given an element λ ∈ l ∞ , one can define 

a linear continuous functional λ(·) :l 1 → R 1 by the equation 

(7.4) λ(x) = ∑ i 

λ i x i ∀x ∈ l 1 , ||λ(·)|| = ||λ|| l∞ . 

It is also known (see, e.g., [52, p. 86]) that any continuous linear functional λ(·) : 

l 1 → R 1 can be presented in the form (7.4) with some λ ∈ l ∞ . Note that from (7.3) 

it follows that (φ 1 (y),φ 2 (y),...) ∈ l 1 and ( ∂φ1 ∂φ 

∂y j 

, 2 

∂y j 

,...) ∈ l 1 for any y ∈ Y , and, 

hence, for any λ =(λ 1 ,λ 2 ,...) ∈ l ∞ , the function ψ λ (y), 

(7.5) ψ λ (y) def 

= ∑ i 

λ i φ i (y), 

is continuously differentiable, with ∇ψ λ (y) = ∑ i λ i∇φ i (y). 

Let us now rewrite problem (1.4) in a “standard” LP form by using the representation 

(7.2). Let M(Y × U) (respectively, M + (Y × U)) stand for the space of 

all (respectively, all nonnegative) measures with bounded variations defined on Borel 

subsets of Y × U, andletA(·) :M(U × Y ) ↦→ R 1 × l 1 stand for the linear operator 

defined for any γ ∈M(U × Y ) by the equation 



(∫ 

∫ 

A(γ) def 

= 1 Y ×U (y, u)γ(dy, du), (∇φ i (y) T f(y, u) 

Y ×U 

Y ×U 

) 

+ C(φ i (y 0 )−φ i (y)))γ(dy, du), i =1, 2,... . 

In this notation problem (1.4) takes the form 

(7.6) min 

γ 

{〈g, γ〉 | A(γ) =(1, 0), γ ∈M + }, 

where 0 is the zero element of l 1 ,and〈·,γ〉, here and in what follows, stands for the 

integral of the corresponding function over γ. 

Define now the linear operator A ∗ (·) :R 1 × l ∞ ↦→ C(Y × U) ⊂M ∗ (Y × U) by 

the equation 

(7.7) 

A ∗ (μ, λ) def 

= μ + ∇ψ λ (·) T f(·, ·)+C(ψ λ (y 0 ) − ψ λ (·)) ∀ μ ∈ R 1 , ∀ λ =(λ i ) ∈ l ∞ , 

where ψ λ (·) is as defined in (7.5). Note that from (7.7) it follows that, for any 

γ ∈M(Y × U), 

∫ 

〈A ∗ (μ, λ),γ〉 = (μ1 Y ×U (y, u)+∇ψ λ (y) T f(y, u)+C(ψ λ (y 0 )−ψ λ (y)))γ(dy, du) 

Y ×U 

(7.8) 

def 

= 〈(μ, λ),A(γ)〉. 

That is, the operator A ∗ (·) istheadjointofA(·), and, hence, the problem dual to 

(7.6) can be written in the form (see page 39 in [2]) 

(7.9) sup 

(μ,λ)∈R 1 ×l ∞ 

{μ |−A ∗ (μ, λ)+g(·) ≥ 0} 

and, by (7.7), is equivalent to 

sup 

(μ,λ)∈R 1 ×l ∞ 

{μ |−μ −∇ψ λ (y) T f(y, u) − C(ψ λ (y 0 ) − ψ λ (y)) 

(7.10) + g(u, y) ≥ 0 ∀(y, u) ∈ Y × U}. 

Due to the approximation property (7.1), the optimal value in (7.10) will be the same 

as in the problem 

sup {μ |−μ−∇ψ(y) T f(y, u)−C(ψ(y 0 )−ψ(y))+g(u, y) ≥ 0 ∀(y, u) ∈ Y ×U} 

(μ,ψ(·))∈R 1 ×C 1 

= sup {μ |μ ≤∇ψ(y) T f(y, u)+C(ψ(y 0 ) − ψ(y)) + g(u, y) ∀(y, u) ∈ Y × U} 

(μ,ψ(·))∈R 1 ×C 1 = μ ⋆ (C, y 0 ), 

the latter being equivalent to (3.1). 

Let us now discuss the possibility of approximating the IDLP problem (1.4) and 

its dual (3.1) with certain finite dimensional LP problems obtained by truncating the 

infinite system of constraints in (7.2) to a system with a finite number of constraints 

and by considering probability measures concentrated on a finite number of points. 



Let the points (y l ,u k ) ∈ Y × U, l =1,...,L Δ , k =1,...,K Δ , define a grid on 

Y ×U parameterized by Δ > 0 in such a way that, for any (y, u) ∈ Y ×U, thereexists 

agridpoint(y l ′,u k ′) satisfying the inequality ||(y, u) − (y l ′,u k ′)|| ≤ aΔ, a=const. 

Let us define the polyhedral set W N,Δ (C, y 0 ) ⊂ R LΔ +K Δ 

by the equation 

{ 

∑ 

W N,Δ (C, y 0 ) def 

= γ = {γ l,k }≥0 : γ l,k =1, ∑ (∇φ i (y l ) T f(y l ,u k ) 

l,k 

l,k 

} 

(7.11) + C(φ i (y 0 ) − φ i (y l )))γ l,k =0 ∀i =1, 2,...,N , 

∑ K 

Δ 

where ∑ def 

l,k 

= ∑ L Δ 

l=1 k=1 and the indexation of the components of γ ∈ W N,Δ (C, y 0 ) 

corresponds to the indexation of the grid points, and let us consider the following finite 

dimensional LP problem (in what follows we call this the (N,Δ)-problem) 

∑ 

(7.12) min γ l,k g(y l ,u k ) def 

= g N,Δ (C, y 0 ). 

γ∈W N,Δ (C,y 0) 

l,k 

Let us also consider the finite dimensional LP problem, which is dual to (7.12), 

{ 

( 

N∑ 

N 

) 

∑ 

max μ : μ ≤ λ i ∇φ i (y l ) T f(y l ,u k )+C λ i (φ 

(μ,λ)∈R 1 ×R } i (y 0 ) − φ i (y l )) 

N 

i=1 

i=1 

(7.13) + g(y l ,u k ) ∀(u l ,y k ) . 

Similarly to [22] and [30] (where the case C = 0 was studied) it can be established 

that, under natural conditions, the following results are true. First (cf. Propositions 

7and9in[30]), 

(7.14) lim lim ρ H(W N,Δ (C, y 0 ),W(C, y 0 )) = 0 

N→∞ Δ→0 

(7.15) ⇒ lim lim 

N→∞ Δ→0 gN,Δ (C, y 0 )=g ∗ (C, y 0 ). 

Second (cf. Proposition 6.3 in [22]), the function 

(7.16) ψ N,Δ (y) def 

= 

N∑ 

i=1 

λ N,Δ 

i φ i (y), 

where λ N,Δ =(λ N,Δ 

i ) is a solution of the finite dimensional dual (7.13), solves problem 

(3.1) approximately in the sense that, for any δ>0, 

μ ∗ (C, y 0 ) − δ ≤ ∇ψ N,Δ (y) T f(y, u)+C(ψ N,Δ (y 0 ) − ψ N,Δ (y)) 

(7.17) + g(y, u) ∀(y, u) ∈ Y × U 

if N is large enough and Δ is small enough. Third, under some additional conditions 

(see Theorem 7.1 in [22]), the feedback control defined by the equation 

(7.18) u N,Δ (y) def 

=argmin 

u∈U {∇ψN,Δ (y) T f(y, u)+g(y, u)} 

is near optimal in (1.3) (in the sense that the value of the objective function obtained 

with this control tends to the optimal one if Δ → 0andN →∞). 

We do not give the proofs of the above mentioned results in the present paper. 

They are very similar to the proofs of the corresponding results of [22] and [30] (also, 

they will be included in a separate publication). However, we will illustrate how these 

results can be used for a construction of near optimal control in (1.3) with a numerical 

example. 



Consider problem (1.3) with the following data (a periodic optimization problem 

with similar data was considered in Example 1 of [30]; see also [6]): 

u ∈ U def 

=[−1, 1] ⊂ R 1 , 

y =(y 1 ,y 2 ) ∈ Y def 

= {(y 1 ,y 2 ) | y 1 ∈ [−5, 5], y 2 ∈ [−8, 8]} ⊂ R 2 ; 

f(y, u) def 

=(y 2 , −4y 1 − 0.3y 2 + u ), 

g(u, y) def 

= u 2 − y1 2 . 

Let us formulate the (N,Δ)-problem (7.12) with the use of the monomials 

φ i1,i 2 

(y 1 ,y 2 ) def 

= y i1 

1 yi2 2 , i 1,i 2 =0, 1,...,K, 

as the functions φ i (y) (note that the number N in (7.11) is equal to (K +1) 2 − 1 

in this case) and with the use of the rectangular grid of size Δ. This problem was 

solved by the CPLEX solver for K =10(N = 120) and Δ = 0.0125, and for C =0.1, 

y 0 =(−3, −5). The optimal value obtained is g N,Δ (C, y 0 ) ≈−3.6014 . Using the 

solution of the dual problem, one can construct the function ψ N,Δ (y) and find the 

feedback control u N,Δ (y) (according to (7.16) and (7.18)). It can be easily seen that 

in this case, u N,Δ (y) allows an explicit representation, 

⎧ 

⎪⎨ 

(7.19) u N,Δ (y) = 

⎪⎩ 

− 1 ∂ψ N,Δ (y 1,y 2) 

2 ∂y 2 

if |− 1 2 

−1 if − 1 2 

1 if − 1 2 

∂ψ N,Δ (y 1,y 2) 

∂y 2 

∂ψ N,Δ (y 1,y 2) 

|≤1, 

∂y 2 

< −1, 

∂ψ N,Δ (y 1,y 2) 

∂y 2 

> 1. 

Substituting this control into system (1.1) and integrating it with MATLAB allows 

one to obtain the state-control trajectory. This trajectory and its projection onto 

the state space (y 1 ,y 2 ) are depicted in Figures 1 and 2. Note that, as can be seen 

from these figures, the trajectories appear to converge to some periodic regime (“limit 

cycle”). 

The value of the objective function numerically evaluated on the state-control 

trajectory is −3.5921, which is close to the value of g N,Δ (α, y 0 ) indicated above (the 

error being of the order of the grid size). Thus, by (7.15), the feedback control defined 

by (7.19) is likely to be near optimal in the problem under consideration. Note that 

the points marked with dots in Figures 1 and 2 are the grid points that correspond 

to the positive (basic) components of the found optimal solution of (N,Δ)-problem 

(7.12) (denote it as γ ∗ = {γl,k ∗ }). This solution can serve as an approximation to a 

solution of the IDLP problem (1.4), which (provided that it is unique) coincides with 

the discounted occupational measure generated by the optimal state-control trajectory 

of (1.3). Hence, the fact that a certain component of γ ∗ is positive indicates 

that the optimal state-control trajectory attends a “small” vicinity of the grid point 

corresponding to this particular component. Consequently, the fact that the trajectories 

depicted in Figures 1 and 2 pass close to the marked points can be interpreted 

as another indication that these trajectories are close to the optimal ones. 

8. Conclusions. We have established the relationships between the optimal control 

problem (1.3) and the IDLP problem (1.4) (Theorem 4.4) based on the duality 

results (Theorem 3.1 and Lemma 3.2). We have shown that the IDLP problem (1.4) 

and its dual can be used for the analysis and solution of the optimal control problem 



1 

0.5 

u 0 

−0.5 

−1 

−6 

−4−2 

0 

2 

4 

6 

y 1 

−8 

−6 

−4 

−2 

0 

y 2 

2 

4 

6 

8 

Fig. 1. Near optimal state-control trajectory. 

8 

6 

4 

2 

y 2 

0 

−2 

−4 

−6 

−8 

−6 −4 −2 0 2 4 6 

y 1 

Fig. 2. Near optimal state trajectory. 

(1.3). In particular we constructed necessary and sufficient optimality conditions for 

this problem (Proposition 4.6), we obtained results characterizing the viability kernel 

of Y (Propositions 5.1 and 5.2), and we indicated a way to use finite dimensional approximations 

of the IDLP problem (1.4) and its dual for finding a numerical solution 

of (1.3) (see section 7). Also, we have shown that the relationships between (1.3) and 

(1.4) can be used for establishing similar relationships between the optimal control 

problem (1.6) (with S →∞) and the IDLP problem (1.7) (Propositions 6.1 and 6.2). 

Appendix. In this section we will prove Theorem 3.1 and give an outline of the 

proof of Lemma 4.3. 

Proof of Theorem 3.1(iii). If the function ψ(·) satisfying (3.4) exists, then 

and, hence, 

min 

(y,u)∈Y ×U {−∇ψ(y)T f(y, u) − C(ψ(y 0 ) − ψ(y))} > 0 

(A.1) 

lim 

α→∞ 

min {g(y, (y,u)∈Y ×U u)+α(−∇ψ(y)T f(y, u) − C(ψ(y 0 ) − ψ(y)))} = ∞. 

This implies that the optimal value of problem (3.1) is unbounded (μ ∗ (C, y 0 )=∞). 

Assume now that the optimal value of problem (3.1) is unbounded. That is, there 

exists a sequence (μ k ,ψ k (·)) such that lim k→∞ μ k = ∞, 



(A.2) μ k ≤ g(y, u)+∇ψ k (y) T f(y, u)+C(ψ k (y 0 ) − ψ k (y)) ∀(y, u) ∈ Y × U 

(A.3) ⇒ 1 ≤ 1 μ k 

g(y, u)+ 1 μ k 

(∇ψ k (y) T f(y, u)+C(ψ k (y 0 )−ψ k (y))) 

∀(y, u) ∈ Y ×U. 

For k large enough, 

1 

μ k 

|g(y, u)| ≤ 1 2 

for all (y, u) ∈ Y × U. Hence 

1 

2 ≤ 1 μ k 

(∇ψ k (y) T f(y, u)+C(ψ k (y 0 ) − ψ k (y))) ∀(y, u) ∈ Y × U. 

That is, the function ψ(y) def 

= − 1 

μ k 

ψ k (y) satisfies (3.4). 

ProofofTheorem3.1(i). From (3.2) it follows that, if W (C, y 0 ) is not empty, then 

the optimal value of problem (3.1) is bounded. 

Conversely, let us assume that the optimal value μ ∗ (C, y 0 )ofproblem(3.1)is 

bounded, and let us establish that W (C, y 0 ) is not empty. Assume that it is not true 

and W (C, y 0 ) is empty. Define the set Q by the equation 

{ 

∫ 

Q def 

= x =(x 1 ,x 2 ,...):x i = (∇φ i (y) T f(y, u)+C(φ i (y 0 ) − φ i (y)))γ(du, dy), 

Y ×U 

} 

(A.4) γ ∈P(Y × U) . 

It is easy to see that the set Q is a convex and compact subset of l 1 (the fact that Q 

is relatively compact in l 1 is implied by (7.3); the fact that it is closed follows from 

P(Y × U) being compact in the weak convergence topology). 

By (7.2), the assumption that W (C, y 0 ) is empty is equivalent to the assumption 

that the set Q does not contain the “zero element” (0 ∉Q). Hence, by a separation 

theorem (see, e.g., [52, p. 59]), there exists ¯λ =(¯λ 1 , ¯λ 2 ,...) ∈ l ∞ such that 

∑ 

0=¯λ(0) > max x∈Q 

¯λ i x i 

=max γ∈P(Y ×U) 

∫ 

Y ×U 

i 

(∇ψ¯λ(y) T f(y, u)+C(ψ¯λ(y 0 ) − ψ¯λ(y)))γ(dy, du) 

=max (y,u)∈Y ×U {∇ψ¯λ(y) T f(y, u)+C(ψ¯λ(y 0 ) − ψ¯λ(y))}, 

where ψ¯λ(y) = ∑ ¯λ i i φ i (y) (see (7.5)). This implies that the function ψ(y) def 

= ψ¯λ(y) 

satisfies (3.4), and, by Theorem 3.1(iii), μ ∗ (C, y 0 ) is unbounded. Thus, we have 

obtained a contradiction that proves that W (C, y 0 )isnotempty. 

Proof of Theorem 3.1(ii). By Theorem 3.1(i), if the optimal value of problem (3.1) 

is bounded, then W is not empty, and hence a solution to problem (1.4) exists. 

Define the set ˆQ ⊂R 1 × l 1 by the equation 

∫ 

ˆQ def 

= {(θ, x) :θ ≥ g(y, u)γ(du, dy), x =(x 1 ,x 2 ,...), 

∫ 

(A.5) x i = 

Y ×U 

Y ×U 

(∇φ i (y) T f(y, u)+C(φ i (y 0 ) − φ i (y)))γ(dy, du), γ ∈P(Y × U)}. 



The set ˆQ is convex and closed. Also, for any j =1, 2,..., the point (θ j , 0) ∉ ˆQ, 

def 

where θ j = g ∗ (C, y 0 ) − 1 j and 0 is the zero element of l 1. On the basis of a separation 

theorem (see [52, p. 59]), one may conclude that there exists a sequence (κ j ,λ j ) ∈ 

R 1 × l ∞ ,j=1, 2,... (with λ j def 

=(λ j 1 ,λj 2 ,...)) such that 

( 

κ j g ∗ (C, y 0 ) − 1 ) 

+ δ j ≤ inf 

j 

(θ,x)∈ ˆQ 

{ 

κ j θ + ∑ i 

λ j i x i 

{ ∫ 

= inf κ j θ + (∇ψ λ j (y) T f(y, u)+C(ψ λ j (y) − ψ λ j (y 0 )))γ(du, dy) 

γ∈P(Y ×U) 

Y ×U 

∫ 

} 

(A.6) 

s.t. θ ≥ g(y, u)γ(du, dy) , 

Y ×U 

where δ j > 0 for all j and ψ λ j (y) = ∑ i λj i φ i(y). From (A.6) it immediately follows 

that κ j ≥ 0. Let us show that κ j > 0. In fact, if it was not the case, one would obtain 

that 

∫ 

0 0. 

Dividing (A.6) by κ j one can obtain that 

g ∗ (C, y 0 ) − 1 ( 

j < g ∗ (C, y 0 ) − 1 ) 

+ δj 

j κ j 

≤ 

{∫ ( 

min 

g(y, u)+ 1 ) } 

γ∈P(Y ×U) Y ×U κ j (∇ψ λ j (y)T f(y, u)+C(ψ λ j (y 0 ) − ψ λ j (y))) γ(dy, du) 

{ 

= min g(y, u)+ 1 } 

(y,u)∈Y ×U 

κ j (∇ψ λ j (y)T f(y, u)+C(ψ λ j (y 0 ) − ψ λ j (y))) ≤ μ ∗ (C, y 0 ) 

⇒ g ∗ (C, y 0 ) ≤ μ ∗ (C, y 0 ). 

The latter and (3.2) prove (3.3). 

Outline of the proof of Lemma 4.3. Let V stands for the set of all Lebesgue 

measurable relaxed controls v(·) :[0+∞) ↦→ P(U), and let V Y δ 

(y 0 ) ⊂Vbe such 

that, for every v(·) ∈V Y δ 

(y 0 ), the solution y(t, y 0 ,v(·)) of (4.16) satisfies the inclusion 

y(t, y 0 ,v(·)) ∈ Y δ for all t>0. 



For any v ′ (·),v ′′ (·) ∈V, define 

∫ 

∫ 

(A.7) α ′ i(t) def 

= φ i (u)v ′ (t, du), α ′′ 

i (t) def 

= 

U 

U 

φ i (u)v ′′ (t, du), 

where φ i (u),i =1, 2,..., is a sequence of Lipschitz continuous functions, which is 

dense in the unit ball of C(U). Let e m (·) :[0,T] → R 1 , m =1, 2,..., be a sequence 

of square integrable functions which is dense in L 2 [0,T], and let 

) 

(A.8) 

¯ζT (α ′ i (·),α′′ 

def 

i (·)) = 

(∣ 

∞∑ ∣∣∣∣ ∫ T 

∫ T 

2 −m e m (t)α ′ i (t)dt − 

m=1 

0 

0 

e m (t)α ′′ 

i (t)dt ∣ ∣∣∣∣ 

∧ 1 

, 

(A.9) 

ζ T (v ′ (·),v ′′ (·)) def 

= 

∞∑ 

i=1 

2 −i ḡ T (α ′ i (·),α′′ i (·)), 

(A.10) 

ζ(v ′ (·),v ′′ (·)) def 

= 

∞∑ 

2 −l ζ l (v ′ (·),v ′′ (·)). 

l=1 

It can be shown that ζ(·, ·) introduced in (A.10) defines a metric on V and that V is 

compact in this metric (see, e.g., Lemma 2.2 in [12]). Also it can be shown that the 

following two statements are valid: (i) V Y δ 

(y 0 ), V Y (y 0 )arecompactand 

(A.11) lim δ→0 V Y δ 

(y 0 ) ⊂V Y (y 0 ); 

(ii) for any continuous q(y, u) :R m × U → R 1 and ¯q(y, u) :R m ×P(U) → R 1 defined 

as in (4.17), the functional 

(A.12) 

Ψ q (v(·)) def 

= 

∫ +∞ 

0 

e −Ct¯q(y(t, y 0 ,v(·)),v(t))dt 

is continuous in v(·). From (A.11) and continuity of Ψ(v(·)) it follows that 

{ 

} 

(A.13) 

min Ψ q (v(·)) ≤ lim δ→0 min Ψ q (v(·)) . 

v(·)∈V Y (y 0) 

By Assumption I, 

v(·)∈V Y δ (y 0) 

(A.14) 

min Ψ g (v(·)) = V 

v(·)∈V Y C Y (y 0), 

(y 0) 

where Ψ g (v(·)) is defined by (A.12) with q(·) =g(·). Since 

one now can use (A.13) to obtain that 

min Ψ g (v(·)) ≤ V Y δ 

v(·)∈V Y C (y 0) ∀ δ>0, 

δ (y 0) 

(A.15) 

V Y C (y 0) ≤ lim δ→0 V Y δ 

C (y 0). 

Since V Y δ 

C (y 0) ≤ V Y C (y 0) ∀δ >0, (A.15) implies (4.13). 



Acknowledgment. We thank S. Rossomakhine for his help with solving the 

numerical example of section 7. 

REFERENCES 

[1] O. Alvarez and M. Bardi, Ergodic problems in differential games, in Advances in Dynamic 

Games Theory, S. Jorgensen, M. Quincampoix, and T.L. Vincent, eds., Birkhäuser Boston, 

Boston, MA, 2007, pp. 131–152. 

[2] E.J. Anderson and P. Nash, Linear Programming in Infinite-Dimensional Spaces, Wiley, 

Chichester, 1987. 

[3] M. Arisawa, H. Ishii, and P.-L. Lions, A characterization of the existence of solutions for 

Hamilton–Jacobi equations in ergodic control problems with applications, Appl. Math. Optim., 

42 (2000), pp 35–50. 

[4] Z. Artstein, Invariant measures and their projections in nonautonomous dynamical systems, 

Stoch. Dyn., 4 (2004), pp. 439–459. 

[5] J.-P. Aubin, Viability Theory, Birkhäuser Boston, Boston, MA, 1991. 

[6] J. Azzato and J.B. Krawczyk, Applying a finite horizon numerical optimization method to 

a periodic optimal control problem, Automatica, 44 (2008), pp. 1642–1651. 

[7] M. Bardi and I. Capuzzo-Dolcetta, Optimal Control and Viscosity Solutions of Hamilton– 

Jacobi–Bellman Equations, Systems Control Found. Appl., Birkhäuser Boston, Boston, 

MA, 1997. 

[8] A. Bensoussan, Perturbation Methods in Optimal Control, John Wiley, New York, 1989. 

[9] D.P. Bertsekas and S.E. Shreve, Stochastic Optimal Control: The Discrete Time Case, 

Academic Press, New York, 1978. 

[10] G.K. Basak, V.S. Borkar, and M.K. Ghosh, Ergodic control of degenerate diffusions, 

Stochastic Anal. Appl., 15 (1997), pp. 1–17. 

[11] A.G. Bhatt and V.S. Borkar, Occupation measures for controlled Markov processes: Characterization 

and optimality, Ann. Probab., 24 (1996), pp. 1531–1562. 

[12] V.S. Borkar and V. Gaitsgory, On averaging of singularly perturbed controlled stochastic 

differential equations, Appl. Math. Optim., 56 (2007), pp. 169–209. 

[13] H. Brézis, Analyse fonctionnelle. Théorie et applications, Collection Mathématiques Appliquées 

pour la Maîtrise, Masson, Paris, 1983. 

[14] P. Cardaliaguet, P. Bettiol, and M. Quincampoix, Zero-sum state constraint differential 

game: Existence of a value for Bolza problem, Internet. J. Game Theory, 34 (2006), pp. 

495–527. 

[15] D.A. Carlson, A.B. Haurie, and A. Leizarowitz, Optimal Control on Infinite Time Horizon, 

2nd ed., Springer-Verlag, Berlin, 1991. 

[16] F.H. Clarke, Yu.S. Ledyaev, R.J. Stern, and P.R. Wolenski, Qualitative properties of 

trajectories of control systems: A survey, J. Dynam. Control Systems, 1 (1995), pp. 1–48. 

[17] F. Colonius and W. Kliemann, The Dynamics of Control, Birkhäuser Boston, Boston, MA, 

2000. 

[18] M. Crandall, H. Ishii, and P.-L. Lions, User’s guide to the viscosity solutions of second 

order partial differential equations, Bull. AMS, 27 (1992), pp. 1–67. 

[19] M.-O. Czarnecki and L.L. Rifford, Approximation and regularization of Lipschitz functions: 

Convergence of the gradients, Trans. Amer. Math. Soc., 358 (2006), pp. 4467–4520. 

[20] T.D. Donchev and A.L. Dontchev, Singular perturbations in infinite-dimensional control 

systems, SIAM J. Control Optim., 42 (2003), pp. 1795–1812. 

[21] L.C. Evans and D. Gomes, Linear programming interpretations of Mather’s variational principle, 

ESIAM Control Optim. Calc. Var., 8 (2002), pp. 693–702 (electronic). 

[22] L. Finlay, V. Gaitsgory, and I. Lebedev, Duality In linear programming problems related 

to deterministic long run average problems of optimal control, SIAM J. Control Optim., 

47 (2008), pp. 1667–1700. 

[23] W.H. Fleming and H.M. Soner, Controlled Markov Processes and Viscosity Solutions, 

Springer-Verlag, New York, 1991. 

[24] W.H. Fleming and D. Vermes, Convex duality approach to the optimal control of diffusions, 

SIAM J. Control Optim., 27 (1989), pp. 1136–1155. 

[25] H. Frankowska, Hamilton–Jacobi equations: Viscosity solutions and generalized gradients, J. 

Math. Anal. Appl., 141 (1989), pp. 21–26. 

[26] H. Frankowska and F. Rampazzo, Filippov’s and Filippov–Wazewski’s theorems on closed 

domains, J. Differential Equations, 161 (2000), pp. 449–478. 



[27] V. Gaitsgory, Suboptimization of singularly perturbed control problems, SIAM J. Control 

Optim., 30 (1992), pp. 1228–1249. 

[28] V. Gaitsgory, On a representation of the limit occupational measures set of a control systems 

with applications to singularly perturbed control systems, SIAM J. Control Optim., 43 

(2004), pp. 325–340. 

[29] V. Gaitsgory, Averaging and near viability of singularly perturbed control systems, J.Convex 

Anal., 13 (2006), pp. 329–352. 

[30] V. Gaitsgory and S. Rossomakhine, Linear programming approach to deterministic long run 

average problems of optimal control, SIAM J. Control Optim., 44 (2006), pp. 2006–2037. 

[31] D. A. Gomes and A. M. Oberman, Computing the effective Hamiltonian using a variational 

approach, SIAM J. Control Optim., 43 (2004), pp. 792–812. 

[32] L. Grüne, On the relation between discounted and average optimal value functions, J.Differential 

Equations, 148 (1998), pp. 65–99. 

[33] L. Grüne, Asymptotic controllability and exponential stabilization of nonlinear control systems 

at singular points, SIAM J. Control Optim., 36 (1998), pp. 1485–1503. 

[34] K. Helmes and R. H. Stockbridge, Numerical comparison of controls and verification of 

optimality for stochastic control problems, J. Optim. Theory Appl., 106 (2000), pp. 107– 

127. 

[35] D. Hernandez-Hernandez, O. Hernandez-Lerma, and M. Taksar, The linear programming 

approach to deterministic optimal control problems, Appl. Math., 24 (1996), pp. 17–33. 

[36] O. Hernandez-Lerma and J.B. Lasserre, Markov Chains and Invariant Probabilities, 

Birkhäuser-Verlag, Basel, 2003. 

[37] M. Kamenskii, P. Nistri, and M. Quincampoix, Singularly perturbed second order differential 

inclusions by an averaging method, Int. J. Pure Appl. Math., 3 (2002), pp. 343–355. 

[38] E. Klein and A.C. Thompson, Theory of Correspondence, Wiley-Interscience, New York, 

1984. 

[39] T.G. Kurtz and R.H. Stockbridge, Existence of Markov controls and characterization of 

optimal Markov controls, SIAM J. Control Optim., 36 (1998), pp. 609–653. 

[40] H.J. Kushner, Weak Convergence Methods and Singularly Perturbed Stochastic Control and 

Filtering Problems, Birkhäuser Boston, Boston, MA, 1990. 

[41] H.J. Kushner and P.G. Dupuis, Numerical Methods for Stochastic Control Problems in Continuous 

Time, 2nd ed., Springer-Verlag, New York, 2001. 

[42] J.B. Lasserre, D. Henrion, C. Prieur, and E. Trélat, Nonlinear optimal control via occupation 

measures and LMI-relaxations, SIAM J. Control Optim., 47 (2008), pp. 1643–1666. 

[43] A. Leizarowitz, Optimal controls for diffusions in R d —A min-max, max-min formula for the 

minimal cost growth rate, J. Math. Anal. Appl., 149 (1990), pp. 180–209. 

[44] A. Leizarowitz and A.J. Zaslavski, Infinite-horizon discrete-time optimal control problems, 

J. Math. Sci., 116 (2003), pp. 3369–3386. 

[45] J.G. Llavona, Approximation of Continuously Differentiable Functions, Math. Stud. 130, 

North–Holland, Amsterdam, The Netherlands, 1986. 

[46] K.R. Parthasarathy, Probability Measures on Metric Spaces, Academic Press, New York, 

1967. 

[47] S. Pickenhain and V. Lykina, Sufficiency conditions for infinite horizon optimal control 

problems, in Recent Advances in Optimization, Lecture Notes in Econom. and Math. Systems 

563, Springer, Berlin, 2006, pp. 217–232. 

[48] M. Quincampoix, Differential inclusions and target problems, SIAM J. Control Optim., 30 

(1992), pp. 324–335. 

[49] M. Quincampoix and C. Rainer, Stochastic control and compatible subsets of constraints, 

Bull. Sci. Math., 129 (2005), pp. 39–55. 

[50] M. Quincampoix and F. Watbled, Averaging method for discontinuous Mayer’s problem of 

singularly perturbed control systems, Nonlinear Anal. Theory Methods Appl., 54 (2003), 

pp. 819–837. 

[51] J.E. Rubio, Control and Optimization. The Linear Treatment of Nonlinear Problems, Manchester 

University Press, Manchester, UK, 1985. 

[52] W. Rudin, Real and Complex Analysis, 3rd ed., McGraw-Hill, New York, 1987. 

[53] T.R. Rockafellar and R. Wets, Variational Analysis, Grundlehren Math. Wiss. 317, 

Springer-Verlag, Berlin, 1998. 

[54] R.H. Stockbridge, Time-average control of Martingale problems. Existence of a stationary 

solution, Ann. Probab., 18 (1990), pp. 190–205. 

[55] R.H. Stockbridge, Time-average control of martingale problems: A linear programming formulation, 

Ann. Probab., 18 (1990), pp. 206–217. 



[56] A. Vigodner, Limits of Singularly Perturbed Control Problems: Dynamical Systems Approach, 

Thesis for the Degree of Doctor of Philosophy, The Weizmann Institute of Science, Rehovot, 

Israel, 1995. 

[57] C. Villani, Topics in Optimal Transportation, Grad. Stud. Math. 58, AMS, Providence, RI, 

2003. 

[58] R. Vinter, Convex duality and nonlinear optimal control, SIAM J. Control Optim., 31 (1993), 

pp. 518–538. 

[59] J. Warga, Optimal Control of Differential and Functional Equations, Academic Press, New 

York, 1972. 

[60] G.G. Yin and Q. Zhang, Continuous-Time Markov Chains and Applications. A Singular 

Perturbation Approach, Springer, New York, 1997.

Linear Programming Approach to Deterministic Infinite Horizon ...

Create successful ePaper yourself

Delete template?

Save as template?