On Averaging of Singularly Perturbed Controlled ... - Study at UniSA

On Averaging of Singularly PerturbedControlled Stochastic Differential EquationsVivek Borkar ∗ † and Vladimir Gaitsgory ‡ §August 10, 2006Abstract. An averaged system to approximate the slow dynamics of a twotimescale nonlinear stochastic control system is introduced. Validity of the approximationis established. Special cases are considered to illustrate the generaltheory.1 IntroductionIn this paper, we consider a system of nonlinear singularly perturbed (SP) controlledstochastic differential equations (CSDE). A small parameter ɛ > 0 is introducedin the system in such a way that the state variables are decomposed intoa group of slow variables, which change their values with rates of the order O(1),and a group of fast ones, which change their values with rates of the order O( 1 ɛ ).Singularly perturbed problems of control and optimization have been studiedintensively in both deterministic and stochastic settings (see [1]-[12], [14], [20]-[25],[30], [31], [33]-[36], [38]-[46], [48], [49], [54]-[58], [61]-[63], [65], [67] and referencestherein for a sample of the literature), with SP CSDE being specifically addressedin [2], [3], [12], [14], [20], [40], [41], [48].Originally, the most common approaches to SP control systems, especially inthe deterministic case, were related to a so-called reduction technique based on∗ V. Borkar is with School of Technology and Computer Science, Tata Institute of FundamentalResearch, Homi Bhabha Rd., Mumbai 400005, India, e-mail: borkar@tifr.res.in† Work partly supported by grant III.5(157)/99-ET from the Department of Scienceand Technology, Govenment of India.‡ V. Gaitsgory is with the School of Mathematics, University of South Australia, MawsonLakes Campus, Mawson Lakes SA, 5095, Australia, e-mail: v.gaitsgory@unisa.edu.au§ The research undertaken by this author was supported by the Australian ResearchCouncil Discovery Grants DP0346099 and DP0664330.1

equating of the small parameter to zero (that is, establishing results in the spiritof Tichonov’s theorem; see [14], [24], [40], [41], [45], [46], [48], [54], [57], [62], [65]).Later, it was realized that, while being very efficient in many important specialclasses of problems, the reduction technique approaches may not lead to a correctapproximation in dealing with general nonlinear SP control systems (see, e.g., [5],[7],[33],[49]).A number of averaging type approaches allowing one to deal with nonlinearSP control systems in which equating of the singular perturbations parameter tozero may not be justifiable were proposed in [1]-[12], [14], [20], [22], [23], [43],[44], [30], [31], [33]-[36], [38], [39], [48], [58], [63]. In purely deterministic settingaveraging techniques based on the idea of approximating the slow dynamics by thesolutions of an averaged system, in which the role of controls is played by limitoccupational measures of the associated system (that is, the system that woulddescribe the fast dynamics if the slow state variables were “frozen”) have beendeveloped in [4], [5], [7]-[10], [34]-[36], [63]. In particular, in [34] (see also [35] and[36]), it has been shown that the controls of the averaged system can be takento be measure valued functions that have their values in the limit occupationalmeasures set (LOMS) of the associated system, the latter being characterized bycertain linear constraints (the proof of the validity of this characterization wasbased on a result in stochastic control obtained in [59]). In a recent paper [20] itwas established that, in case the fast dynamics is independent of the slow statevariables, a similar averaging technique is applicable in dealing with SP controlledstochastic differential equations, with the LOMS of the associated system beingin this case equal to the set of marginal probability distributions of the relaxedstationary solutions of this system (the structure of the latter set was described in[15] and [59]). In this paper we extend the results of [20] to the general case whenthe fast dynamics involves the dependence on slow state variables.The paper is organized as follows. In Section 2, the definitions of the associatedand averaged systems along with some notations are introduced, and also somekey preliminary results are established. In Section 3, it is established that, under amild condition, any partial limit in law of any solution of the SP CSDE is a weaksolution of the averaged system (Theorem 3.3). In Section 4, more restrictiveassumptions are introduced and more strong convergence results are presented(Theorem 4.2). Also in this section, two important related results (Proposition 4.1and 4.2) are given. In Section 5, special cases are discussed. Sections 6,7 and 8contain the proofs of the results presented in Section 3,4 and 5 respectively. Oneof the results of Section 2 (namely, Lemma 2.2(ii)) is also proved in section 5.2

2 Notation and preliminariesThroughout this paper, for a Polish space S, C(S) and C b (S) will stand for thespaces of continuous and bounded continuous functions on S respectively andP(S) will denote the Polish space of probability measures on S with the weakconvergence topology ([18], Chapter 2). Also, for an S−valued random variableX, L(X) will denote its law, viewed as an element of P(S). For a deterministicor stochastic process X(t) (t ≥ 0), X(·) will denote its entire trajectory and X(I)will stand for its trajectory restricted to the time interval I = [a, b]. Each of theseis viewed as an element of an appropriate function space which will be clear fromthe context.Let ɛ > 0 be a small parameter. Consider the R d × R s −valued process(z ɛ (·), x ɛ (·)) described by the system of SP CSDEHere:dz ɛ (t) = h(z ɛ (t), x ɛ (t), u ɛ (t))dt + γ(z ɛ (t), x ɛ (t), u ɛ (t))dB(t), (1)dx ɛ (t) = 1 ɛ m(zɛ (t), x ɛ (t), u ɛ (t))dt + 1 √ ɛσ(z ɛ (t), x ɛ (t), u ɛ (t))dW (t). (2)• For a prescribed compact metric ‘action’ space A, h : R d × R s × A → R d ,γ : R d × R s × A → R d×d , m : R d × R s × A → R s , σ : R d × R s × A → R s×sare Lipschitz in the first two arguments uniformly w.r.t. the third one,• The initial values are fixed: (z ɛ (0), x ɛ (0)) = (z 0 , x 0 ),• B(·), W (·) are resp. d− and s−dimensional independent standard Brownianmotions,• u ɛ (·) is an A−valued control process with measurable paths satisfying thenonanticipativity condition: for ∀ t ′ ≥ t, (B(t ′ ) − B(t), W (t ′ ) − W (t)) isdefindependent of the σ−field F t = the completion of∩ θ>t σ(z ɛ (y), x ɛ (y), B(y), W (y), u ɛ (y), y ≤ θ). (3)Note that, in absence of control (the case of uncontrolled dynamics), singularlyperturbed stochastic differential equations similar to (1)-(2) have been studied in[42] and [44], with [42] being probably the first publication devoted to the genuineanalysis of such systems.The triplet (z ɛ (t), x ɛ (t), u ɛ (t)) will be referred to as a solution and u ɛ (t) as anadmissible control of the SP CSDE (1)-(2). The solutions of (1)-(2) are interpreted3

and the solutions of the associated system (6) satisfy the inequalitysup τ,u(·) E[||x(τ)|| 4 ] < ∞, (9)where in the first case the supremum is over ɛ ∈ (0, ¯ɛ] (¯ɛ > 0 is small enough), overt > 0 and over all admissible controls for SP CSDE (1)-(2), and in the second caseit is over τ > 0 and over all admissible controls u(·) for the associated system (6).Remark: Assumption 1 implies that {L(x ɛ (t)) : t ≥ 0, u ɛ (·) admissible forSP CSDE (1) − (2) } and {L(x(τ)) : τ ≥ 0, u(·) admissible for (6) } are tightin P(R s ).Note that the intuition under Assumption 1 is that there is a sufficiently strongdrift towards the origin when the state moves far away, so as to keep the probabilitymass concentrated enough to keep the fourth moment bounded. A simple sufficientcondition for Assumption 1 to be satisfied is that there exist a twice a continuouslydifferentiable function V (x) : R s → R 1 and positive constants a i , i = 1, ..., 5, suchthata 1 ||x|| 4 ≤ V (x) ≤ a 2 ||x|| 4 + a 3andLV (z, x, u) ≤ −a 4 V (x) + a 5 .Here and in the sequel, L stands for the infinitesimal operator of the associatedsystem. That is, for any twice continuously differentiable functions f(x) : R s →R 1 ,Lf(z, x, u) def= 1 2 tr(σ(z, x, u)σ(z, x, u)T ∇ 2 f(x))+〈∇f(x), m(z, x, u)〉 ∀f ∈ C 2 0(R s ).(10)To verify the validity of (9) under the above condition (the verification of (8)follows similar lines), it is enough to observe that, by Ito formula, for any solution(x(·), u(·)) of (6),ddτ E[V (x(τ))] = E[LV (z, x(τ), u(τ))] ≤ −a 4E[V (x(τ))] + a 5⇒ E[V (x(τ))] ≤ E[V (x(0))] + a 5a 4⇒ a 1 E[||x(τ)|| 4 ] ≤ E[V (x(0))] + a 5a 4≤ a 2 E[||x(0)|| 4 ] + a 3 + a 5a 4< ∞.Note that this condition is the counterpart of geometric ergodicity of Meyn-Tweedie introduced for the discrete time setting in [51].5

Let C0 2(Rs ) be the space of twice continuously differentiable functions f(x) :R s → R 1 which vanish at infinity along with their first and second derivatives andlet D be a countable dense set in C0 2(Rs ). Define the set of probability measuresD z ⊂ P(R s × A) by the equation∫D z = {µ ∈ P(R s × A) : Lf(z, x, u)µ(dxdu) = 0 ∀ f ∈ D}. (11)This set is known to represent the marginal distributions of all stationary relaxedsolutions of the associated system (see Theorem 2.1 in [15] and Theorem 4.1 in[59]). Also, under Assumption 1, the set D z contains all partial limits of occupationalmeasures generated by the solutions of the associated system as the thetime horizon it is considered on tends to infinity (see Theorems 4.1, 4.2 in [20],and Proposition 5.1 in Section 5 below). The latter implies, in particular, thatD z ≠ ∅ ∀z ∈ R d . (12)Lemma 2.1 The set-valued map z → D z is closed and convex valued and it is alsoupper semicontinuous.Proof. It is easy to see that D z is convex and closed in P(R s × A) for each z.The fact that it is upper semicontinuous follows from the fact that the relation∫ Lf(z, x, u)µ(dxdu) = 0 is preserved under convergence of (z, µ) in R d ×P(R s ×A).✷Let us introduce now the averaged system. To do this, let us define the functions¯h : R d × P(R s × A) → R d , and ¯γ : R d × P(R s × A) → R d×d by the equations:∫¯h(z, µ) = h(z, x, u)µ(dxdu), (13)¯γ(z, µ) =√ ∫γ(z, x, u)γ(z, x, u) T µ(dxdu), (14)where the square-root is assumed to be Lipschitz in z uniformly w.r.t. µ. (SeeTheorem 5.2.3 of [60], p. 132, for some sufficient conditions for this to be the case.)The averaged system is the CSDEwhere:dz(t) = ¯h(z(t), µ(t))dt + ¯γ(z(t), µ(t))dB ′ (t), (15)µ(t) ∈ D z(t) ∀t, (16)• z(0) = z 0 (same initial condition as that for the z−component of the solutionof the SP CSDE (1)-(2)),6

• B ′ (·) is a standard d-dimensional Brownian motion,• µ(·) is a P(R s ×A)−valued control process with measurable paths satisfying(16) and also satisfying the nonanticipativity condition: for ∀t 2 ≥ t 1 , B ′ (t 2 )−B ′ (t 1 ) is independent of the completion of the σ−field∩ t ′ >t 1σ(z(t), B ′ (t), µ(t), t ≤ t ′ ).The pairs (z(t), µ(t)) of random processes which satisfy the above conditions willbe called solutions of the averaged system.In the rest of this section we introduce a space of measure valued functionswhich is used in our further consideration. Let ¯R s denote the one point compactificationof R s (see, e.g., [32], p.126). Then any probability measure µ on R s ×A maybe identified with the unique probability measure on ¯R s × A that restricts to µ onR s ×A and perforce assigns zero probability to its complement. The space of probabilitymeasures P( ¯R s × A) is endowed with the weak convergence topology whichis metrizable and compact. Note that the sequence µ k ∈ P(R s × A), k = 1, 2, ...,converges in this topology to µ ∈ P(R s × A) if and only if∫lim f(x, u)µ k (dxdu) =k→∞ R s ×A∫R s ×Af(x, u)µ(dxdu)∀f ∈ C b (R s × A).Let U T and U denote the spaces of measurable functions which map [0, T ] and[0, ∞) respectively into P( ¯R s × A). Let these spaces be topologized as follows: U Thas the coarsest topology that makes the mapsψ f,g (µ([0, T ]) def=∫ T0∫g(t) f(x, u)µ(t, dxdu)dt : U T → R (17)¯R s ×Acontinuous for all f ∈ C( ¯R s × A), g ∈ L 2 [0, T ] and U has the coarsest topologydefwhich is required to make the mapping of µ(·) ∈ U to its truncation µ(·)| [0,T ] =µ([0, T ]) ∈ U T continuous for every T > 0.Lemma 2.2 (i) The spaces U T , T > 0 and U are metrizable; (ii) The spacesU T , T > 0, and U are compact and hence Polish.Proof. Let us show that the spaces U T and U are metrizable by constructing themetrics which are consistent with the topologies of these spaces. Let µ ′ (·), µ ′′ (·)be arbitrary elements of U T and letα ′ i(t) def=∫f i (x, u)µ ′ (t, dxdu) , α¯R s i ′′ (t) def=×A∫f i (x, u)µ ′′ (t, dxdu) , (18)¯R s ×A7

where f i (x, u), i = 1, 2, ..., is a sequence of Lipschitz continuous functions which isdense in the unit ball of C( ¯R s × A). Let e m (·) : [0, T ] → R 1 , m = 1, 2, ..., be asequence of square integrable functions which is dense in L 2 [0, T ] and letḡ T (α i([0, ′ T ]), α i ′′ ([0, T ])) def=Takeg T (µ ′ ([0, T ]), µ ′′ ([0, T ])) def=∞∑m=1∞∑i=1∫ T∫ T2 −m | e m (t)α i(t)dt ′ − e m (t)α i ′′ (t)dt| ∧ 1).00(19)2 −i ḡ T (α ′ i([0, T ]), α ′′i ([0, T ])) ∀µ ′ (·), µ ′′ (·) ∈ U T .(20)It is easy to verify that g T is a metric on U T and that it is consistent with thecoarsest topology which makes the maps (17) continuous.Define now the metric g on U by the equationg(µ ′ (·), µ ′′ (·)) def=∞∑2 −l g l (µ ′ ([0, l]), µ ′′ ([0, l])) ∀µ ′ (·), µ ′′ (·) ∈ U , (21)l=1where µ ′ ([0, l]), µ ′′ ([0, l]) are truncations of µ ′ (·) and respectively µ ′′ (·) to the intervals[0, l], l = 1, 2, ... . The metric g is consistent with the coarsest topology thatmakes the mapping of µ(·) to its truncation µ([0, T ]) continuous for every T > 0.This proves Lemma 2.2 (i). The proof of Lemma 2.2 (ii) is given in Section 8. ✷Lemma 2.3 If µ n ([0, T ]) → µ([0, T ]) in U T for some T > 0, then∫ T ∫∫ T ∫f(t, x, u)µ n (t, dxdu)dt → f(t, x, u)µ(t, dxdu)dt (22)0 ¯R s ×A0 ¯R s ×Afor any f ∈ C([0, T ] × ¯R s × A).Proof. Let the sequence f i (x, u), i = 1, 2, ..., be as above (that is, countable anddense in the unit ball of C( ¯R s ×A)). Then, by definition of the topology of U T , (22)is true if f(t, x, u) = g(t)f i (x, u) for some i and some g ∈ C[0, T ]. By the density of{f i (·)}, it is also true if f(t, x, u) = g(t)q(x, u) for g as above and q(·) ∈ C( ¯R s ×A),hence for linear combinations of such functions. By the Stone-Weirstrass theorem(see, e.g., [32], p. 133), the latter are dense in C([0, T ] × ¯R s × A) and, hence,following standard approximation argument, one can establish that (22) is validfor f ∈ C([0, T ] × ¯R s × A).✷Corollary 2.4 If µ n ([0, T ]) → µ([0, T ]) in U T and µ n (t, R d × A) = µ(t, R d × A) =1 ∀t ∈ [0, T ], then∫ T ∫∫ T ∫f(t, x, u)µ n (t, dxdu)dt → f(t, y, u)µ(t, dxdu)dt (23)0R s ×A80R s ×A

for any f ∈ C b ([0, T ] × R s × A).Proof. Consider ˜µ n def= 1 T µn (t, dxdu)dt and ˜µ def= 1 Tµ(t, dxdu)dt as probabilitymeasures on [0, T ] × ¯R d × A, with ˜µ n ([0, T ] × R d × A) = ˜µ([0, T ] × R d × A) = 1.Then, by Corollary 2.3, for any f ∈ C([0, T ] × ¯R s × A),∫∫f(t, x, u)˜µ n (dtdxdu) →f(t, x, u)˜µ(dtdxdu).[0,T ]× ¯R s ×A[0,T ]× ¯R s ×AThis implies that ˜µ n converges weakly to ˜µ in P([0, T ]× ¯R s ×A). The claim followsnow from Theorem 2.1.1 in [18] (the argument used in proving the equivalence ofthe statements (i), (ii) and (iii) in this theorem is applicable).✷Corollary 2.5 If µ n (·) → µ(·) in U and µ n (t, R d × A) = µ(t, R d × A) = 1 ∀t ∈[0, ∞), then∫ ∞ ∫∫ ∞ ∫e −rt f(t, x, u)µ n (t, dxdu)dt →e −rt f(t, x, u)µ(t, dxdu)dt0R s ×A0R s ×Afor any f ∈ C b ([0, ∞) × R s × A), where r is a positive constant.Proof. By definition of the topology of U, the convergence µ n (·) → µ(·) in Uimplies the convergence µ n ([0, T ]) → µ([0, T ]) in U T for any T > 0. Hence, (23)with f(t, x, u) replaced by e −rt f(t, x, u) is valid for any T > 0. By a routineargument this can be shown to imply (24).✷3 Convergence in law to the solutions of theaveraged systemLet (z ɛ (t), x ɛ (t), u ɛ (t)) be a solution of the SP CSDE (1)-(2) and let µ ɛ (t, dxdu) def=δ (x ɛ (t),u ɛ (t))(dxdu) (i.e., the Dirac measure at (x ɛ (t), u ɛ (t))). Equation (1) can thenbe equivalently (as far as the solutions’ laws are concerned) rewritten in the form(24)dz ɛ (t) = ¯h(z ɛ (t), µ ɛ (t))dt + ¯γ(z ɛ (t), µ ɛ (t))dB(t), (25)where ¯h(·), ¯γ(·) are defined by (13)-(14) (see, e.g., Corollary 5.1.3 of [60]). The pair(z ɛ ([0, T ]), µ ɛ ([0, T ])) can be considered as a C([0, T ]; R d ) × U T − valued randomvariable, while the pair (z ɛ (·), µ ɛ (·)) can be considered as a C([0, ∞); R d )×U− valuedrandom variable, where U T , U are as introduced in Section 2 and C([0, T ]; R d )(resp. C([0, ∞); R d )) are the spaces of continuous vector functions taking values inR d and defined on [0, T ] (resp. [0, ∞)). The topology of C([0, T ]; R d ) is that of uniformconvergence and the topology of C([0, ∞); R d ) is that of uniform convergenceon compact subsets of [0, ∞).9

Proposition 3.1 Let Assumption 1 be satisfied. Then the set{L(z ɛ ([0, T ]), µ ɛ ([0, T ])), ɛ > 0} ⊂ P(C([0, T ]; R d ) × U T )is relatively compact for any T > 0.Proof. By Prohorov’s theorem ([18], p. 25), it suffices to prove that{L(z ɛ ([0, T ]), µ ɛ ([0, T ])), ɛ > 0} is tight in P(C([0, T ]; R d ) × U T ). Since U T is compact,it is enough to prove that {L(z ɛ ([0, T ])), ɛ > 0} is a tight set in P(C([0, T ]; R d )).The Lipschitz condition in the first two variables of h, γ, m, σ implies at mostlinear growth in these variables. Thus, using the estimates of Lemma 4.12 in [50],p.125, for moments of stochastic integrals, one can establish that there exists aconstant K T such that∫ tmaxt∈[0,T ] E[||zɛ (t)|| 4 ] ≤ K T (E[||z ɛ (0)|| 4 ] + (1 + E[||z ɛ (θ)|| 4 + ||x ɛ (θ)|| 4 ]dθ)0Taking into account Assumption 1 and applying Gronwall-Bellman lemma one canfurther obtain that there exists a constant ¯K T such thatsup t∈[0,T ] E[||z ɛ (t)|| 4 ] ≤ ¯K T (1 + E[||z ɛ (0)|| 4 ]).This, in turn (again, via using Lemma 4.12 of [50]), leads to the existence of aconstant ¯KT such thatE[||z ɛ (t) − z ɛ (θ)|| 4 ] ≤ ¯KT |t − θ| 2 , (26)which is sufficient for the set {L(z ɛ ([0, T ])), ɛ > 0} to be tight in P(C([0, T ]; R d ))(see Theorem 12.2 in [16], p.95).✷Corollary 3.2 L{(z ɛ (·), µ ɛ (·)), ɛ > 0} is relatively compact in P(C([0, ∞); R d ) ×U).Proof. Let (z ɛ(n) (·), µ ɛ(n) (·)) be a sequence of solutions of (25), with ɛ(n) , n =1, 2, ..., tending to zero when n tends to infinity. Using Proposition 3.1 and thediagonalization argument, one can establish that there exists a subsequence, say,n l , such that L(z ɛ(nl) ([0, i]), µ ɛ(nl) ([0, i])) converges (as n l tends to infinity) to anelement of P(C([0, i]; R d ) × U i ) for any i = 1, 2, ... . This implies the convergenceof L(z ɛ(nl) (·), µ ɛ(nl) (·)) in P(C([0, ∞); R d ) × U).✷From the fact that the set L{(z ɛ (·), µ ɛ (·)), ɛ > 0} is relatively compact inP(C([0, ∞); R d ) × U) it follows that the set L{(z ɛ (·), µ ɛ (·), B(·), W (·)), ɛ > 0} isrelatively compact in P(C([0, ∞); R d ) × U × C([0, ∞); R d ) × C([0, ∞); R s )).10

Let η be a limit point of L(z ɛ (·), µ ɛ (·), B(·), W (·)). That is, there exists asequence of ɛ(n) tending to zero as n tends to infinity such thatlimn→∞ L(zɛ(n) (·), µ ɛ(n) (·), B(·), W (·)) def= η ,where η is an element of P(C([0, ∞); R d ) × U × C([0, ∞); R d ) × C([0, ∞); R s )).One can interpret η as the lawη = L(z(·), µ(·), B ′ (·), W ′ (·))for a canonically realized random process (z(·), µ(·), B ′ (·), W ′ (·)), which takes valuesinC([0, ∞); R d ) × U × C([0, ∞); R d ) × C([0, ∞); R s )).Such a process will be referred to as a partial limit-in-law of (z ɛ (·), µ ɛ (·), B(·), W (·)).Note that, by construction the B ′ (·)− and W ′ (·)− components of the partial limitin-laware standard independent d− and s− dimensional Brownian motions.Theorem 3.3 Let Assumption 1 be satisfied and let (z ɛ (t), x ɛ (t), u ɛ (t)) be a solutionof the SP CSDE (1)-(2). Let also µ ɛ (t, dxdu) = δ (x ɛ (t),u ɛ (t))(dxdu) (as above) and(z(·), µ(·), B ′ (·), W ′ (·)) be a partial limit-in-law of (z ɛ (·), µ ɛ (·), B(·), W (·)). Thenthe pair (z(·), µ(·)) is a solution of the averaged system (15)-(16).Proof is in Section 6.Consider the optimal control problem∫ Tinf E[ e −rt k(z ɛ (t), x ɛ (t), u ɛ (t))dt] def= J(z ɛ (·),x ɛ (·),u ɛ ɛ ∗ (27)(·)) 0where k(z, x, u) ∈ C b (R d × R s × A) and the inf is either over the solutions of thesystem (1)-(2) on a finite time interval [0, T ] (in this case we take T def= T andr ≥ 0) or over the solution of this system on the interval [0, ∞) (in which caseT def= ∞ and r > 0).Consider also the optimal control problem∫ Tinf E[ e −rt¯k(z(t), defµ(t))dt] = J0 ∗ , (28)(z(·),µ(·)) 0where ¯k(z, µ) def= ∫ k(z, x, u)µ(dx, du) and the inf is over the solutions of the averagedsystem (15)-(16) on a finite or infinite time horizon (as above).Corollary 3.4 Under the conditions of Theorem 3.3,lim infɛ→0J ∗ ɛ ≥ J ∗ 0 . (29)11

Proof. Let ɛ = ɛ(n) ↓ 0 and the sequence of solutions (z ɛ(n) (·), x ɛ(n) (·), u ɛ(n) (·)) of(1)-(2) be such that∫ Tlim E[ e −rt k(z ɛ(n) (t), x ɛ(n) (t), u ɛ(n) (t))dt] = lim inf Jɛ∗ ɛ(n)→0 0ɛ→0∫ T⇒ lim E[ e −rt¯k(z ɛ(n) (t), µ ɛ(n) (t))dt] = lim inf Jɛ ∗ ,ɛ(n)→0 0ɛ→0where µ ɛ (t, dxdu) = δ (x ɛ (t),u ɛ (t))(dxdu). By Proposition 3.1 and Corollary 3.2, onemay assume, without loss of generality, that(z ɛ(n) (·), µ ɛ(n) (·), B(·), W (·)) → (z(·), µ(·), B ′ (·), W ′ (·))) in law. (30)From Corollaries 2.4 and 2.5 it follows that the map ∫ T0 e−rt¯k(z(t), µ(t))dt is continuousin (z(·), µ(·)). As it is also bounded, the convergence in law (30) impliesthe convergence of the mathematical expectations∫ Tlim E[ e −rt¯k(z ∫ Tɛ(n) (t), µ ɛ(n) (t))dt] = E[ e −rt¯k(z(t), µ(t))dt]ɛ(n)→0 00which, in turn, implies∫ TE[ e −rt¯k(z(t), µ(t))dt] = lim inf Jɛ ∗ .0ɛ→0Since, by Theorem 3.3, (z(·), µ(·)) is a solution of the averaged system (15)-(16),This proves the claim.∫ TE[ e −rt¯k(z(t), µ(t))dt] ≥ J∗0 .0Remark: Note that lim inf ɛ→0 Jɛ∗ may depend on the initial values of the faststate variables x ɛ (0) = x 0 , in which case the inequality (29) may be strict. Thiscan happen, e.g., if the state space of the fast dynamics R s is decomposed intosubsets in such a way that the trajectories of SP CSDE having initial values offast components in one subset can not reach the states with fast components fromanother subset (with whatever admissible controls are used). Conditions ensuringthat the limit of Jɛ ∗ exists and is equal to J0 ∗ are discussed in Sections 4 and 5below.✷12

4 Further assumptions and convergence resultsIn this section we will assume thatand that the averaged system is of the formγ(z, x, u) def= γ(z). (31)dz(t) = ¯h(z(t), µ(t))dt + γ(z(t))dB(t) , µ(t) ∈ D z(t) ∀t (32)(that is, in particular, it is considered with the same Brownian motion as in (1)).Also, we will use more structured nonanticipativity conditions. Namely, we willassume (as in [20]) that admissible controls in the CSDE (1)-(2) and in the averagedsystem (32) are progressively measurable with respect to a given (the same in bothcases) continuous and complete filtration { ˆF t } of σ-fields such that:• {B(θ), W (θ); θ ≤ t} is measurable with respect to ˆF t for t ≥ 0 ,• For t ′ ≥ t ≥ 0 , B(t ′ ) − B(t) and W (t ′ ) − W (t) are independent of ˆF t .The admissible controls in the associated system (6) will be assumed to be progressivelymeasurable with respect to F τ = ˆF ɛτ .Let us define the metric ρ(·, ·) on P( ¯R s × A) by the equationρ(µ ′ , µ ′′ ) def=∞∑(2c i ) −i |〈f i , µ ′ 〉 − 〈f i , µ ′′ 〉| ∀µ ′ , µ ′′ ∈ P( ¯R s × A) , (33)i=1where f i (x, u), i = 1, 2, ..., is a sequence of Lipschitz continuous functions which isdense in the unit ball of C( ¯R s × A), with c i ≥ 1 being Lipschitz constants of f i ,and here and in what follows 〈f, µ〉 def= ∫ f(x, u)µ(dxdu). Note that this metric isconsistent with the weak convergence topology of P( ¯R s × A).Assumption 2: Corresponding to arbitrary random z and x, which are independentof W ′ (·) and satisfy the inequalities E[||z|| 4 ] ≤ c , E[||x|| 4 ] ≤ c (c is apositive constant), and corresponding to an arbitrary random µ ∈ D z , which isindependent of W ′ (·), there exists a solution (x(·), u(·)) of the associated system(6) such that x(0) = x andE[ρ(µ, µ S )] ≤ ν c (S) ,lim ν c(S) = 0 , (34)S→∞Here µ S is the occupational measure generated by (x(·), u(·)) on the interval [0, S].This measure is defined by taking µ S (Q) def= 1 Smeas{ τ ∈ [0, S] : (x(τ), u(τ)) ∈ Q}13

for any Borel subset Q of ¯R s × A, with meas standing for the Lebesgue measureon [0, S] (see, e.g., [20]). Equivalently, µ S can be defined by the equations:∫f i (x, u)µ S (dxdu) = 1 ∫ Sf i (x(τ), u(τ))dτ, i = 1, 2, ... , (35)¯R s ×A S0where f i (·) are as in (33). The notation ν c (S) in (34) is to reflect the fact that thisestimate is supposed to be uniform in z, x and µ but it may depend on the boundc.Note that Assumption 2 is implied by a certain “mixing” condition stated in theproposition below. The proof of the proposition is based on results of [20], where itis also shown that, under some additional provisions, this mixing condition is notonly sufficient but also necessary for Assumption 2 to be true (see Corollary 4.3 anRemark 3 in [20]). An important special case in which the mixing condition and,thus, Assumption 2 are satisfied is discussed in Section 5 below (see Assumption4, Proposition 5.2 and subsequent comments).Proposition 4.1 Letσ(z, x, u) def= σ(z, x) (36)and let Assumption 1 be satisfied. Then Assumption 2 is satisfied if the associatedsystem (6) has the following property: for any initial condition x(0) = x ′ 0 andadmissible control u ′ (·), corresponding to any other initial condition x(0) = x ′′0there exists an admissible control u ′′ (·) such that the solutions x ′ (·) and x ′′ (·) of(6) (obtained with x ′ 0 , u′ (·) and x ′′0 , u′′ (·) respectively) satisfy the inequalities∫ SE[ || 1 f i (x ′ (τ), u ′ (τ))dτ − 1 f i (x ′′ (τ), u ′′ (τ))dτ|| ]S 0S 0≤ ν i (S)(1 + E[||x ′ 0|| 4 ] + E[||x ′′0|| 4 ] + E[||z|| 4 ]) i = 1, 2, ... , (37)where ν i (·) are monotone decreasing functions such that lim S→∞ ν i (S) = 0.Proof. In case z is fixed (non-random), the proof follows from Corollary 4.3 in[20]. The extension to the case when z is random and satisfies (7) is straightforward.✷To introduce our next assumption, let us define the metric β(·, ·) on R d ×P( ¯R s × A) by the equation∫ Sβ((v ′ , µ ′ ), (v ′′ , µ ′′ )) def= ||v ′ − v ′′ || + ρ(µ ′ , µ ′′ ) (38)∀(v ′ , µ ′ ), (v ′′ , µ ′′ ) ∈ R d × P( ¯R s × A), where here and in the sequel || · || is theEuclidean norm in the finite dimensional space. Define also a multivaled mapQ zdef= {(v, µ) | v = ¯h(z, µ), µ ∈ D z } ⊂ R d × P(R s × A) . (39)14

Note that, under Assumptions 1 and 2, the map z → Q z is convex- and compactvalued.Assumption 3 (Lipschitz continuity of Q z ): For any z ′ , z ′′ ∈ R d ,β H (Q z ′, Q z ′′) ≤ c ∗ ||z ′ − z ′′ || , (40)where β H (·, ·) is the Hausdorff metric defined on the closed and bounded subsetsof R d × P( ¯R s × A) by the metric β(·, ·) and c ∗ is a suitable constant.Remark: Assumption 3 is implied by the Lipschitz continuity in z of ¯h(z, µ)(which, in turn, is implied by the Lipschitz continuity of h(z, x, u) in z) in caseD z is independent of z, that is, if the associated system (6) does not involve thedependence on z. An important special case when this assumption (as well asAssumption 2) are satisfied in the presence of such a dependence, is considered inthe next section.Theorem 4.2 Let Assumptions 1-3 be satisfied. Then, corresponding to any solution(z(·), µ(·)) of the averaged system (32), there exists a solution (z ɛ (·), x ɛ (·),u ɛ (·)) of the SP CSDE (1)-(2) such that, for any T > 0lim max E[ɛ→0 t∈[0,T ] ||zɛ (t) − z(t)|| 2 ] = 0 ,lim E[g T (µ ɛ ([0, T ]), µ([0, T ]))] = 0 , (41)ɛ→0where µ ɛ (t, dxdu) def= δ (x ɛ (t),u ɛ (t))(dxdu) and g T (·, ·) is defined in (20).Proof. The theorem is established by Lemma 7.3 and Lemma 7.4 (see section 7).✷Remark: Theorem 4.2 is a generalization of Theorem 5.1 (ii) in [20], where thecorresponding result was obtained for the case when the associated system is independentof z. Note that a statement generalizing Theorem 5.1 (i) in [20] can beestablished too. Namely, it can be shown that under conditions 1-3, correspondingto any solution (z ɛ (·), x ɛ (·), u ɛ (·)) of the SP CSDE (1)-(2), there exists a solution(˜z ɛ (·), ˜µ ɛ (·)) of the averaged system (32) such that, for any T > 0,lim max E[ɛ→0 t∈[0,T ] ||˜zɛ (t) − z ɛ (t)|| 2 ] = 0 ,lim E[g T (˜µ ɛ ([0, T ]), µ ɛ ([0, T ]))] = 0 . (42)ɛ→0The proof of this statement is similar to the proof of Theorem 5.1 (i) in [20] andwe do not include it in the paper.Corollary 4.3 Under Assumptions 1-3, corresponding to any solution (z(·), µ(·))of the averaged system (15)-(16), there exists a solution (z ɛ (·), x ɛ (·), u ɛ (·)) of theSP CSDE (1)-(2) such thatlimɛ→0 L(zɛ (·), µ ɛ (·)) = L(z(·), µ(·)). (43)15

Also,lim J ɛ ∗ = J0 ∗ . (44)ɛ→0Proof. The fact that (41) is valid for any T > 0 implies thatlim ɛ→0 L(z ɛ ([0, T ]), µ ɛ ([0, T )) = L(z([0, T ]), µ([0, T ])) is valid for any T > 0, whichin turn implies the validity of (43).Let (z ∗ (t), µ ∗ (t)) be a solution of the averaged system (15)-(16) such that theminimal value in (28) is achieved:∫ TE[ e −rt¯k(z ∗ (t), µ ∗ (t))dt] = J0 ∗ . (45)0By (43), there exists a solution (z ∗ɛ (t), x ∗ɛ (t), u ∗ɛ (t)) of the SP CSDE (1)-(2) suchthatlimɛ→0 L(z∗ɛ (·), µ ∗ɛ (·)) = L(z ∗ (·), µ ∗ (·)), (46)where µ ∗ɛ (t, dxdu) def= δ (x ∗ɛ (t),u ∗ɛ (t))(dxdu). Similarly to Corollary 3.4 one can showthat from (45) and (46) it follows that∫ T∫ Tlim E[ e −rt k(z ∗ɛ (t), x ∗ɛ (t), u ∗ɛ (t))dt] = lim E[ e −rt¯k(z ∗ɛ (t), µ ∗ɛ (t))dt] = J0 ∗ .ɛ→0 0ɛ→0 0(47)Since Jɛ ∗ ≤ E[ ∫ T0 e−rt k(z ∗ɛ (t), x ∗ɛ (t), u ∗ɛ (t))dt] , from (47) in turn follows thatlim sup ɛ→0 Jɛ ∗ ≤ J0 ∗ . The latter and (29) imply (44).✷Remark: By (44) and (47),∫ TE[ e −rt k(z ∗ɛ (t), x ∗ɛ (t), u ∗ɛ (t))dt] = Jɛ ∗ + ν(ɛ),0lim ν(ɛ) = 0. (48)ɛ→0Hence, knowing a solution of the averaged system which is optimal in (28), one canconstruct a solution of the SP CSDE which is near optimal in (27) (in the sensethat (48) is valid). The procedure of construction of the latter, provided that theformer is known, is described in Section 7.Let us conclude this section with the statement that may provide a furtherinsight in the nature of Assumption 2. Assume that (36) is satisfied and considera “relaxed” version of the associated system (6) defined by the equationdx(τ) = ¯m(z, x(τ), v(τ))dτ + σ(z, x(τ))dW ′ (τ), (49)where ¯m(z, x, v) def= ∫ A m(z, x, u)v(du) ∀(z, x, v) ∈ Rd × R s × P(A) . Note thatwe consider system (49) only with fixed (non-random) z. A pair (x(·), v(·)) will16

e called a relaxed solution of the associated system (or a solution of the relaxedassociated system) if it satisfies (49) with v(·) being a P(A)-valued control process,which is assumed to have measurable paths and to satisfy the nonanticipativitycondition: for ∀ τ 2 ≥ τ 1 , W ′ (τ 2 ) − W ′ (τ 1 ) is independent of the completion of theσ−field ∩ τ ′ >τ 1σ(x(τ), W ′ (τ), v(τ), τ ≤ τ ′ ).Assumption 1 ′ : Every stationary solution of the relaxed associated system (49)satisfies the inequalitysup E[||x(τ)|| 4 ] < c, (50)τ≥0where c is a constant (c can be different for different values of z).Proposition 4.4 Let (36) be true and let Assumption 1 ′ be satisfied. Then, correspondingto any extreme point µ of D z , there exists an ergodic stationary solution(x(·), v(·)) of the relaxed associated system such thatlim ρ(µ, µ S) = 0, a.s., (51)S→∞where µ S is the occupational measure defined on the interval [0, S] by the equations∫¯R s ×Af i (x, u)µ S (dxdu) = 1 Sf i (·) being as in (33).∫ S ∫0Af i (x(τ), u)v(τ)(du)dτ, i = 1, 2, ... ,Proof. The proposition is proved in Section 7.✷5 Special cases and commentsDenote by M(z, S, x) a collection of occupational measures of the associated system,with x and z being as in Assumption 2. Then (34) becomes equivalent tosupµ∈D zinf E[ρ(µ,µ ′ ∈M(z,S,x) µ′ )] ≤ ν c (S) , lim ν c(S) = 0 . (52)S→∞That is, Assumption 2 postulates a kind of lower semicontinuity property ofM(z, S, x), which is established (under the corresponding conditions) by proposition4.1. The upper semicontinuity property of M(z, S, x) is established by thefollowing propositionProposition 5.1 Let Assumption 1 be satisfied and z be fixed (non-random). Thensup E[ρ(µ, D z )] ≤ ν c (S) ,µ∈M(z,S,x)lim ν c(S) = 0 . (53)S→∞17

where ρ(µ, D z ) def= inf µ ′ ∈D zρ(µ, µ ′ ) . If, in addition, Assumption 1 ′ is satisfied,then, for any Lipschitz continuous vector function q(x, u) : R s × A → R j (j ≥ 1),the set Vzqandsup E[dist(µ∈M(z,S,x)def= {v : v = ∫ R s ×A q(x, u)µ(dxdu) , µ ∈ D z} is convex and compact∫R s ×Aq(x, u)µ(dxdu) , V qz ) ] ≤ ν q c (S) ,limS→∞ νq c (S) = 0 , (54)where dist(v, V qz ) def= inf v ′ ∈V qz ||v − v′ || ∀v ∈ R j .Proof. The proof follows from Theorem 4.2 and Corollary 4.4 in [20].✷Remark: By Theorem 2.1 in [15] (or Theorem 4.1 in [59]), the set D z coincideswith the set of one dimensional marginal distributions corresponding to allstationary solutions of the relaxed associated system (49). Hence, Assumption 1 ′is equivalent to the assumption that∫sup ||x|| 4 µ(dxdu) ≤ c , (55)µ∈D zwhich implies that the set D z is compact and that∫||x||µ(dxdu) ≤ ν(N) ∀µ ∈ D z (56)||x||≥Nfor some function ν(N) tending to zero as n → ∞.Using Proposition 5.1, one can establish that the following exponential stabilityassumption is sufficient for Assumptions 2 and 3 to be valid.Assumption 4: For any admissible control u(·), the solutions x ′ (·) and x ′′ (·) of(6) obtained with the initial conditions x ′ (0) = x ′ 0 and x′′ (0) = x ′′0 satisfy theinequalitylimτ→∞ E[||x′ (τ) − x ′′ (τ)||] ≤ ae −bτ E[||x ′ 0 − x ′′0||] ∀τ ≥ 0 , (57)where a, b are some positive constants.Proposition 5.2 Let Assumptions 1 and 1 ′ be valid and (36) be satisfied. ThenAssumptions 2 and 3 are valid if Assumption 4 is valid.Proof. The fact that Assumption 2 is valid follows from Proposition 4.1 (since(37) is implied by (57) with u ′ (·) = u ′′ (·)). The validity of Assumption 3 is establishedin Section 8.✷18

Note that Liapunov-type sufficient condition for Assumption 4 to be true can befound in [13], p. 5. In particular, this assumption is valid for purely deterministicsystem (6) (σ(z, x, u) ≡ 0; see, e.g., [35]) if there exist positive definite matricesM 1 and M 2 such that, ∀z ∈ R d , ∀u ∈ A, ∀x ′ , x ′′ ∈ R s ,(m(z, x ′ , u) − m(z, x ′′ , u)) T M 1 (x ′ − x ′′ ) ≤ −(x ′ − x ′′ ) T M 2 (x ′ − x ′′ ). (58)A different condition that leads to the fulfillment of Assumption 2 is that thesmallest eigenvalue of σ(z, x)σ(z, x) T is bounded away from zero uniformly in z, x.The validity of Assumption 2 in this case can be verified (although we do not do itin the paper) on the basis of results of [15] and [47], which give a characterizationof D z using Markov controls, i.e., controls that depend on the present state alone.Propositions 4.1 and 5.1 allow one to interpret D z as a limit of the set ofoccupational measures of system (6), or using the terminology of [20], the “limitoccupational measures set” of system (6). In the deterministic setting resultssimilar to Propositions 4.1, 5.1 and to Theorem 4.2 have been obtained in [34](see also closely related earlier results in [35] and [36] and some details aboutthe comparison of the deterministic and stochastic cases in Section 5 of [20]).Important developments in averaging of deterministic singularly perturbed controlsystems over limit occupational measures of the associated system can be foundin [4] - [7], [9], [10], [63].In [36] it was shown that, if a deterministic singularly perturbed control systemis linear in fast variables and controls, then only the first moments of the measuresfrom the LOMS enter the averaged system and this system proves to be equivalentto a so-called reduced system obtained via equating of the small parameter tozero. Let us outline how a similar result can be established in the stochasticcontrol framework. Assume that the control set A is a convex and compact subsetof R k and that representation (31) as well asandh(z, x, u) = F 1 (z)x + H 1 (z)u (59)m(z, x, u) = F 2 (z)x + H 2 (z)u , σ(z, x, u) = σ(z) (60)are valid, with F 1 (·), H 1 (·) and F 2 (·), H 2 (·) being matrix functions of the correspondingdimensions. Assume that ||e F 2(z)τ || ≤ ae −bτ (that is, the eigenvalues ofF 2 (z) have real parts which are less than some negative number for all z ∈ R d ),which leads to the validity of Assumption 4.By (59), the averaged system (15) is reduced todz(t) = h(z(t), ¯x(t), ū(t))dt + γ(z(t))dB(t), (¯x(t), ū(t)) ∈ Ω z(t) , (61)19

where Ω z is the set of the first moments corresponding to the probability measuresfrom D z :∫defΩ z = {(¯x, ū) : (¯x, ū) = (x, u)µ(dxdu) , µ ∈ D z } . (62)Proposition 5.3 Let (60) be valid and the associated system satisfy Assumptions1 and 1 ′ . Then the following representation for Ω z is valid:Ω z = {(¯x, ū) : ¯x = −F −12 (z)H 2 (z)ū , ū ∈ A} . (63)Proof. The proof is in Section 8.Using (63), one can show that the system (61) is equivalent to the systemdz(t) = h(z(t), −F −12 (z)H 2 (z)ū(t), ū(t))dt + γ(z(t))dB(t), ū(t) ∈ A . (64)Note that this system can be obtained if one: (i) multiplies (2) by ɛ; (ii) equatesɛ in the resulting equation to zero and expresses x as the function of z and u; (iii)substitutes thus obtained expression for y into (1).If, in addition, the function k(z, x, u) in (27) is convex in (x, u), then one canshow also that the ”averaged” optimal control problem (27) is reduced to theproblem∫ Tinf E[ e −rt k(z(t), −F2 −1 (z(t))H 2 (z(t))ū(t), ū(t))dt] = J0 ∗ , (65)(z(·),ū(·)) 0where inf is over the solutions of (64). Note that singularly perturbed stochasticcontrol systems with the structure similar to that defined by (59) and (60) havebeen studied in [40],[41], where results concerning the approximation of the solutionsof the SP systems by the solutions of (64) were established using completelydifferent approach. For the case when the associated system is independent of z astatement similar to Proposition 5.3 was given (without a proof) in [20]Another interesting special case to mention is that when the slow system (2)does not involve the Brownian motion in its description. That is, γ(·) ≡ 0. Inthis case the averaged system (32) is purely deterministic and, by Theorem 4.2,one can restrict the optimization in (27) to ”almost deterministic” solutions of(1)-(2) (despite of the presence of the stochastic elements in the fast dynamics).A similar situation was studied in [1] where the fast dynamics was defined by acontrolled Markov chain making transitions in time intervals of the length ɛ. Notethat the state and action spaces of this chain were finite and that the approach ofthis paper can be applied to obtain results similar to [1] for the chains with infinite(countable or uncountable) state and action spaces.20

6 Proof of Theorem 3.3Theorem 3.3 is established by the following two lemmas.Lemma 6.1 (z(·), µ(·)) satisfies (15) a.s. and µ(·) is nonanticipative.Lemma 6.2 µ(·) satisfies (16) a.s.Proof of Lemma 6.1. For f ∈ Cb 2(Rd ) def= {f : R d → R is twice continuouslydifferentiable and f, ∂f ∂∂x i,2 f∂x i ∂x j, 1 ≤ i, j ≤ d, are bounded }, defineG µ f(z) = 1 2 tr[( ∫∫+〈γ(z, x, u)γ T (z, y, u)µ(dxdu))∇ 2 f(z)]h(z, x, u)µ(dxdu), ∇f(z)〉= 1 2 tr[¯γ2 (z, µ)∇ 2 f(z)] + 〈¯h(z, µ), ∇f(z)〉,where, by definition (see (14)), the matrix ¯γ(·) is symmetric.Let∫Mf ɛ (t) deft= f(z ɛ (t)) − f(z ɛ (0)) − G µ ɛ (y)f(z ɛ (y))dy, t ≥ 0. (66)0Applying the Ito formula, one can readily verify that Mf ɛ (t) as well as the function∫ t(Mf ɛ (t)) 2 −0||¯γ(z ɛ (θ), µ ɛ (θ))∇f(z ɛ (θ))|| 2 dθ, t ≥ 0,are martingales w.r.t. F t , where || · || stands for the Euclidean norm. Also, usingthe same approach, one can show that the functions∫ tMf ɛ (t)B i (t) −0d∑j=1¯γ ji (z ɛ (θ), µ ɛ (θ)) ∂f∂z j(z(θ))dθ, 1 ≤ i ≤ d, t ≥ 0,are martingales with respect to F t , where B i (·) is the i th component of B(·) and¯γ ji (·) is the ji−th entry of the matrix ¯γ(·).Let nowM f (t) def= f(z(t)) − f(z(0)) −It can be shown that M f (t),∫ t0G µ(θ) f(z(θ))dθ, t ≥ 0. (67)∫ t(M f (t)) 2 − ||¯γ(z(θ), µ(θ))∇f(z(θ))|| 2 dθ, t ≥ 0, (68)021

and∫ tM f (t)B i(t) ′ −0d∑j=1are martingales with respect to the filtration F ′ t¯γ ji (z(θ), µ(θ)) ∂f∂z j(z(θ))dθ, 1 ≤ i ≤ d, t ≥ 0, (69)def= the completion of∩ s>t σ(z(θ), µ(θ), B ′ (θ), W ′ (θ); θ ≤ s).Let us verify that M f (t) is a martingale w.r.t. F ′ t (the fact that the rest of thefunctions in (68)-(69) are martingales w.r.t. F ′ t is established in a similar way).By definition of M ɛ f (t) (see (66) above) and since it is a martingale w.r.t. F t,∫ tE[(f(z ɛ (t)) − f(z ɛ (θ ′ )) − G µ ɛ (y)f(z ɛ (y))dy)θ ′g(z ɛ ([0, θ]), µ ɛ ([0, θ]), B([0, θ]), W ([0, θ]))] = 0for all t ≥ θ ′ > θ, f ∈ C 2 b (Rd ) and for all continuous bounded g : C([0, θ]; R d ) ×U θ × C([0, θ]; R d ) × C([0, θ]; R s )) → R 1 . Let ɛ = ɛ(n) ↓ 0 be such that(z ɛ(n) (·), µ ɛ(n) (·), B(·), W (·)) → (z(·), µ(·), B ′ (·), W ′ (·))in law. Then passing to the limit in the above equation, one obtains∫ tE[(f(z(t)) − f(z(θ ′ )) − G µ(y) f(z(y))dy)θ ′g(z([0, θ]), µ([0, θ]), B ′ ([0, θ]), W ′ ([0, θ]))] = 0,which by a standard monotone class argument implies that M f (t) is a martingalew.r.t. F ′ t.Given martingales X(t), Y (t), let 〈X〉(t), 〈Y 〉(t) stand for their quadratic variations(see e.g. [47], p 79) and〈X, Y 〉(t) def= 1 (〈X + Y 〉(t) − 〈X〉(t) − 〈Y 〉(t)).2X(t) and Y (t) are called orthogonal if 〈X, Y 〉(t) = 0, t ≥ 0 a.s. Note that ifX(t) and Y (t) are orthogonal, then 〈X + Y 〉(t) = 〈X〉(t) + 〈Y 〉(t) a.s. and that if〈X〉(t) = 0, t ≥ 0 a.s., then X(t) = const , t ≥ 0 a.s.From the fact that the functions in (68)-(69) are martingales w.r.t. F t ′ andfrom Theorems 5.1 and 5.2 in [50], pp.152-153, it follows that a.s.〈M f 〉(t) =∫ t0||¯γ(z(θ), µ(θ))∇f(z(θ))|| 2 dθ , t ≥ 0, (70)22

∫ t〈M f , B i〉(t) ′ =0d∑j=1¯γ ji (z(θ), µ(θ)) ∂f∂z j(z(θ))dθ, 1 ≤ i ≤ d, t ≥ 0. (71)By (71) and Theorem 5.4 in [50], p.160, the following representation is valid a.s.:where ζ(t) is defined by the equationζ(t)def==M f (t) = ζ(t) + ξ(t), t ≥ 0, (72)∫ t d∑ d∑( ¯γ ji (z(θ), µ(θ)) ∂f (z(θ))dB0∂zi(θ)′i=1 j=1j∫ t(∇f(z(θ))) T ¯γ(z(θ), µ(θ))dB ′ (θ) (73)0(B ′ i (·), i = 1, ...d, are the components of B′ (·)) and ξ(t) is a continuous squareintegrablezero mean martingale w.r.t. F ′ t which is orthogonal to ζ(t): 〈ξ, ζ〉(t) = 0.From this orthogonality it follows that a.s.〈M f 〉(t) = 〈ζ〉(t) + 〈ξ〉(t),where, using (73), one can show (see [50], pp. 152-160) that a.s.〈ζ〉(t) =∫ t0||¯γ(z(θ), µ(θ))∇f(z(θ))|| 2 dθ, t ≥ 0.The comparison of the last two expressions with (70) allows one to conclude thata.s.〈ξ〉(t) = 0, t ≥ 0 ⇒ ξ(t) = const t ≥ 0,where the constant is zero because ξ(·) is zero mean. Substituting the latter and(73),(70) into (67), one can obtain that a.s.∫ tf(z(t)) = f(z(0)) + G µ(θ) f(z(θ))dθ0∫ t+ (∇f(z(θ))) T ¯γ(z(θ), µ(θ))dB ′ (θ), t ≥ 0.0Since the above expression is valid for any f(z) from C 2 b (Rd ) it can be easily verifiedthat the pair (z(t), µ(t)) satisfies (15) for t ≥ 0 a.s.Let us next show that µ(·) is nonanticipative. By definition of the admissiblecontrols for the SP CSDE (1)-(2),E[g 1 (z ɛ ([0, θ]), µ ɛ ([0, θ]), B([0, θ]), W ([0, θ]) g 2 (B(t) − B(θ ′ ), W (t) − W (θ ′ ))]= E[g 1 (z ɛ ([0, θ]), µ ɛ ([0, θ]), B([0, θ]), W ([0, θ])] E[g 2 (B(t) − B(θ ′ ), W (t) − W (θ ′ ))]23

for any t ≥ θ ′ > θ and for any continuous bounded g 1 : C([0, θ]; R d ) × U θ ×C([0, θ]; R d ) × C([0, θ]; R s )) → R 1 and g 2 : R d × R s → R 1 . Choosing a sequence ofɛ = ɛ(n) ↓ 0 as before and passing to the limit in the above expression along thissequence, one obtainsE[g 1 (z([0, θ]), µ([0, θ]), B ′ ([0, θ]), W ′ ([0, θ])g 2 (B ′ (t) − B ′ (θ ′ ), W ′ (t) − W ′ (θ ′ ))]= E[g 1 (z([0, θ]), µ([0, θ]), B ′ ([0, θ]), W ′ ([0, θ])]E[g 2 (B ′ (t) − B ′ (θ ′ ), W ′ (t) − W ′ (θ ′ ))].The latter implies that, for any θ ′ > θ, (B ′ (t) − B ′ (θ ′ ), W ′ (t) − W ′ (θ ′ )) is independentof∩ θ>s σ(z(y), µ(y), B ′ (y), W ′ (y), y ∈ [0, θ])Letting θ ′ ↓ s, it follows that so is (B ′ (t) − B ′ (s), W ′ (t) − W ′ (s)) (by continuity ofthe Brownian motion). This implies the required nonanticipativity of µ(·). ✷Proof of Lemma 6.2. For f ∈ D (where D is the same as in (11) ), extendLf : R d × R s × A → R to Lf : R d × ¯R s × A → R by setting Lf(·, ∞, ·) ≡ 0. Now,for t > θ,ɛ(f(x ɛ (t)) − f(x ɛ (θ))) =∫ t ∫θ+ √ ɛ¯R s ×A∫ tθLf(z ɛ (y), x, u)µ ɛ (y, dxdu)dy〈∇f(x ɛ (y)), σ(z ɛ (y), x ɛ (y), u ɛ (y))dW (y)〉.As ɛ → 0, the l.h.s. tends to zero. The variance of the second term in the r.h.s.is of the order of O(ɛ) and, hence, it also tends to zero in law. Consequently,the first term in the r.h.s. must tend to zero in law. By definition of (z(·), µ(·)),there exists a sequence of values of ɛ(n), n = 1, 2, ..., tending to zero such that(z ɛ(n) (·), µ ɛ(n) (·)) → (z(·), µ(·)) in law as S def= C([0, ∞); R d ) × U−valued randomvariables. Applying Skorohod’s theorem (see e.g. [18], p. 23), one can concludefrom this that there exist S−valued random variables (˜z n (·), ˜µ n (·)), n ≥ 1,(˜z(·), ˜µ(·)) defined on a common probability space such that they agree in law with(z ɛ(n) (·), µ ɛ(n) (·)), n ≥ 1, (z(·), µ(·)) resp. and such that (˜z n (·), ˜µ n (·)) → (˜z(·), ˜µ(·))a.s. From the definition of the topology in U and Lemma 2.3 it follows that a.s.∫ t ∫θLf(˜z n (y), x, u)˜µ n (y, dxdu)dy →¯R s ×A∫ t ∫Hence, the r.h.s. of the above expression, and therefore∫ t ∫θLf(z(y), x, u)µ(y, dxdu)dy¯R s ×AθLf(˜z(y), x, u)˜µ(y, dxdu)dy.¯R s ×A24

must be a.s. zero. As this is true for all t > θ, the integrand is zero a.e., where theLebesgue null set may be taken to be the same for all f ∈ D. Hence, by taking asuitable version of µ(·), we may drop the qualification ’a.e.’ and obtain that a.s.∫Lf(z(y), x, u)µ(y, dxdu) = 0 ∀y ∈ [0, ∞).¯R s ×AThe claim would now follow if we prove that µ(y)(R s × A) = 1 for all y. For t > θ,define µ n θ,t ∈ P( ¯R s × A) in such a way that∫f(x, u)µ n θ,t(dxdu) = 1¯R s ×At − θ∫ t ∫θf(x, u)µ ɛ(n) (y, dxdu)dy¯R s ×Afor ∀f ∈ C( ¯R s × A). Likewise, define µ θ,t ∈ P( ¯R s × A) so that∫f(x, u)µ θ,t (dx, du) = 1¯R s ×At − θ∫ t ∫for ∀f ∈ C( ¯R s × A).From the fact that µ ɛ(n) (·) → µ(·) in law it follows that→ E[∫ TE[0∫ T0∫g(y)∫g(y)for any g ∈ L 2 [0, T ] and any T ≥ 0. Hence,θf(x, u)µ(y, dxdu)dy¯R s ×Af(x, u)µ ɛ(n) (y, dxdu)dy]¯R s ×Af(x, u)µ(y, dxdu)dy]¯R s ×A∫∫E[ f(x, u)µ n θ,t(dxdu)] → E[ f(x, u)µ θ,t (dx, du)] ∀f ∈ C( ¯R s × A)¯R s ×A¯R s ×Awhich implies that µ n θ,t → µ θ,t in law as P( ¯R s × A)−valued random variables.By Assumption 1, for any η > 0 there exists a compact K η ⊂ R s such thatP (x ɛ (t) ∈ K η ) ≥ 1 − η for all t ≥ 0 and all admissible u ɛ (·). By definition,E[µ ɛ(n) (y, K η × A)] = P (x ɛ (y) ∈ K η )⇒E[µ n θ,t(K η × A)]def=∫1 tP (x ɛ(n) (y) ∈ K η )dy ≥ 1 − η ∀n. (74)t − θ θUsing the Skorohod’s theorem as above and by a small abuse of terminology, wereplace the convergence µ n θ,t → µ θ,t ‘in law’ by that ‘a.s.’ and, thus, obtainE[µ θ,t (K η × A)] ≥ E[lim sup µ n θ,t(K η × A)] ≥ 1 − η ⇒n→∞25

whereE[µ θ,t (R s × A)] = 1 ⇒ µ θ,t (R s × A) = 1 a. s. , (75)µ θ,t (K η × A) def=∫1 tµ(y, K η × A)dyt − θ θThe second equality in (75) is true for all t > θ rational outside a common zeroprobability set, hence by continuity, for all t > θ. It follows that almost surely,µ(y, R s × A) = 1 for a.e. y, where the ‘a.e.’ may be dropped as before by takinga suitable version of µ(·).✷7 Proofs for Section 4In this section, K > 0 will stand for a general constant which may vary from oneusage to another.Let the functions f i (x, u), i = 1, 2, ..., be as in (33) and¯ρ(µ ′ , µ ′′ ) def= (Define ˆρ(·, ·) and ˆβ(·, ·) by the equations:∞∑(2c i ) −2i |〈f i , µ ′ 〉 − 〈f i , µ ′′ 〉| 2 ) 1 2 .i=1ˆρ(µ ′ , µ ′′ ) def= ρ(µ ′ , µ ′′ ) + ¯ρ(µ ′ , µ ′′ ) ∀µ ′ , µ ′′ ∈ P(R s × A) , (76)ˆβ((v ′ , µ ′ ), (v ′′ , µ ′′ )) def= β((v ′ , µ ′ ), (v ′′ , µ ′′ )) + ˆρ(µ ′ , µ ′′ ) (77)∀(v ′ µ ′ ), (v ′′ , µ ′′ ) ∈ R d × P( ¯R s × A). Since ¯ρ(µ ′ , µ ′′ ) ≤ Kρ(µ ′ , µ ′′ ), one can obtainthatρ(µ ′ , µ ′′ ) ≤ ˆρ(µ ′ , µ ′′ ) ≤ (1 + K)ρ(µ ′ , µ ′′ )andβ((v ′ , µ ′ ), (v ′′ , µ ′′ )) ≤ ˆβ((v ′ , µ ′ ), (v ′′ , µ ′′ )) ≤ (2 + K)β((v ′ , µ ′ ), (v ′′ , µ ′′ )).The latter implies thatˆβ H (Q z ′, Q z ′′) ≤ K||z ′ − z ′′ || ∀z ′ , z ′′ ∈ R d , (78)where ˆβ H (·, ·) is the Hausdorff metric defined on the subsets of R d × P( ¯R s × A)by the metric ˆβ(·, ·).Lemma 7.1 For any (v, µ) ∈ R d × P( ¯R s × A) and z ∈ R d there exists a uniquesolution of the problemmin ˆβ((v, µ), (v ′ , µ ′ )). (79)(v ′ ,µ ′ )∈Q z26

Proof follows from the fact that Q z is compact and convex and from thatˆβ((v, µ), (1 − λ)(v ′ , µ ′ ) + λ(v ′′ , µ ′′ ))< (1 − λ) ˆβ((v, µ), (v ′ , µ ′ )) + λ ˆβ((v, µ), (v ′′ , µ ′′ )) (80)∀λ ∈ (0, 1), ∀(v ′ , µ ′ ) ≠ (v ′′ , µ ′′ ). ✷The element of Q z which attains minimum in (79) will be referred to as theprojection of (v, µ) onto Q z . It will be denoted as Ψ(z, v, µ), with its first andsecond components being denoted as ψ 1 (z, v, µ) and ψ 2 (z, v, µ) respectively. Notethat, by definition of Q z ,ψ 1 (z, v, µ) = ¯h(z, ψ 2 (z, v, µ)) (81)and that, by Lemma 6.1 and (78), the map Ψ(z, v, µ) and, hence, ψ 1 (z, v, µ) andψ 2 (z, v, µ) are continuous.Let now (z(t), µ(t)) be a solution of the averaged system (15)-(16) on [0, ∞)and v(t) def= ¯h(z(t), µ(t)). Let us partition the time interval with the points t l ,where ∆(ɛ) > 0 is a function of ɛ such thatDenote:˜θ ldef= (ṽ l , ˜µ l ) def= 1∆(ɛ)t ldef= l∆(ɛ), l = 0, 1, ... , (82)lim ∆(ɛ) = 0, limɛ→0∫ tlt l−1(v(t), µ(t))dt ,where ¯v ldef= ψ 1 (z(t l ), ˜θ l ) , ¯µ ldef= ψ 2 (z(t l ), ˜θ l ) and, by (81),∆(ɛ)= ∞ . (83)ɛ→0 ɛ¯θldef= Ψ(z(t l ), ˜θ l ) = (¯v l , ¯µ l ) ,¯v l = ¯h(z(t l ), ¯µ l ). (84)Lemma 7.2 Let T be an arbitrary positive and N (ɛ,T )def= ⌊ Tpart ofT∆(ɛ) . Then, for any l = 1, 2, ..., N (ɛ,T ) ,∆(ɛ) ⌋be the integerE[ ||ṽ l − ¯v l || 2 ] + E[ ˆρ 2 (˜µ l , ¯µ l ) ] ≤ E[ ˆβ 2 (˜θ l , ¯θ l ) ] ≤ K∆(ɛ). (85)Proof. The left inequality in (85) follows from the definition of ˆβ. To prove theright inequality, note that from the fact that (v(t), µ(t)) def= θ(t) ∈ Q z(t) and from(16) and (78) it follows that, for all t ∈ [t l−1 , t l ],ˆβ(θ(t), Q z(tl )) ≤ ˆβ(θ(t), Q z(t) ) + ˆβ H (Q z(t) , Q z(tl )) ≤ K||z(t) − z(t l )||,27

where ˆβ(θ, Q z ) def= min θ ′ ∈Q zˆβ(θ, θ ′ ). Since, as can be easily verified,ˆβ((1 − λ)θ ′ + λθ ′′ , Q z ) ≤ (1 − λ) ˆβ(θ ′ , Q z ) + λ ˆβ(θ ′′ , Q z ), ∀λ ∈ (0, 1)∀θ ′ , θ ′′ ∈ R d × P( ¯R s × A), one can obtain thatˆβ(˜θ l , ¯θ l ) = ˆβ(˜θ l , Q z(tl )) ≤ 1∆(ɛ)∫ tlt l−1ˆβ(θ(t), Qz(tl ))dt ≤K∆(ɛ)∫ tlt l−1||z(t) − z(t l )||dt.⇒ ˆβ 2 (˜θ l , ¯θ∫l ) ≤ K2 tl∆ 2 (ɛ) ( ||z(t) − z(t l )||dt) 2 ≤ K2t l−1∆(ɛ)Similarly to (26), one can establish that∫ tlE[||z(t) − z(θ)|| 4 ] ≤ K T |t − θ| 2t l−1||z(t) − z(t l )|| 2 dt (86)⇒ E[||z(t) − z(θ)|| 2 ] ≤ √ K T |t − θ| (87)⇒ max t∈[tl−1 ,t l ]E[||z(t) − z(t l )|| 2 ] ≤ √ K T ∆(ɛ), (88)where K T is an appropriate constant. The estimates (86),(88) imply (85).Let us now construct a solution (z ɛ (t), x ɛ (t), u ɛ (t)) of the CSDE (1)-(2) whichwill satisfy (43). Initially, let us take u ɛ (t) = u ∀t ∈ [0, t 1 ), where u is an arbitraryelement of A. Let (z ɛ (t), x ɛ (t)) be the solution of (1)-(2) obtained with this control.Define:def¯θ 1 = Ψ(z ɛ (t 1 ), ¯θdef1 ) , ¯v 1 = ψ 1 (z ɛ (t 1 ), ¯θdef1 ) , ¯µ 1 = ψ 2 (z ɛ (t 1 ), ¯θ 1 ) .Note that||¯v 1 − ¯v l || + ˆρ(¯µ 1 , ¯µ 1 ) = ˆβ(¯θ1 , ¯θ 1 ) =min ˆβ(θ, ¯θ 1 )θ∈Q z ɛ (t1 )≤ ˆβ H (Q z ɛ (t 1 ), Q z(t1 )) ≤ K||z ɛ (t 1 ) − z(t 1 )||. (89)Consider the associated system (6) with z = z ɛ (t 1 ), W ′ (τ) = W (ɛτ)−W √ (t 1)ɛandwith the initial condition x(0) = x ɛ (t 1 ). By Assumption 2, there exists a solutionx 1 (τ), u 1 (τ) of this system such that∫ Sɛ∫E[ |(S ɛ ) −1 f i (x 1 (τ), u 1 (τ))dτ − f i (x, u)¯µ 1 (dx, du)| ] ≤ ν i (ɛ), (90)0deffor any f i (x, u) used in the definition of the metric (33), where S ɛ = ∆(ɛ)ɛ→ ∞and ν i (ɛ) → 0 as ɛ → 0. Also, similarly to Corollary 4.5 in [20] it can be shown(using Assumptions 1 and 2) that∫ Sɛ∫E[ ||(S ɛ ) −1 h(z ɛ (t 1 ), x 1 (τ), u 1 (τ))dτ − h(z ɛ (t 1 ), x, u)¯µ 1 (dx, du)|| ] 2 ≤ ν(ɛ),028✷(91)

where as ɛ → 0. Note that in (90),(91) and in the sequel, E[ · ] 2 stands for (E[ · ]) 2 .Define u ɛ (t) def= u 1 ( t−t 1ɛ) ∀t ∈ [t 1 , t 2 ). Let (z ɛ (t), x ɛ (t)) be the solution of (1)-(2) obtained with this control. Define also x ɛ def1 (t) = x( t−t 1ɛ). Note that the lattersatisfies the equationdx ɛ 1(t) = 1 ɛ h(zɛ (t 1 ), x ɛ 1(t), u ɛ (t))dt + 1 √ ɛσ(z ɛ (t 1 ), x ɛ 1(t), u ɛ (t))dW (t) (92)which is obtained via the replacement of the time scale t = ɛτ in (6). For t ∈ [t 1 , t 2 ],we haveE[||x ɛ 1(t) − x ɛ (t)|| 2 ≤2ɛ 2 E[|| ∫ tt 1(h(z ɛ (t 1 ), x ɛ 1(s), u ɛ (s)) − h(z ɛ (s), x ɛ (s), u ɛ (s))ds|| 2 ]≤≤≤+ 2 ∫ tɛ E[|| (σ(z ɛ (t 1 ), x ɛ 1(s), u ɛ (s)) − σ(z ɛ (s), x ɛ (s), u ɛ (s))dW (s)|| 2 ]t 1 2∆(ɛ) ∫ tɛ 2 E[ ||h(z ɛ (t 1 ), x ɛ 1(s), u ɛ (s)) − h(z ɛ (s), x ɛ (s), u ɛ (s))|| 2 ds]t 1+ 2K ∫ tɛ E[ ||σ(z ɛ (t 1 ), x ɛ 1(s), u ɛ (s)) − σ(z ɛ (s), x ɛ (s), u ɛ (s))|| 2 ds]t 1 8∆(ɛ) ∫ t∫ tɛ 2 Clip[2 E[||z ɛ (t 1 ) − z ɛ (s)|| 2 ]ds + E[||x ɛ 1(s) − x ɛ (s)|| 2 ]ds]t 1 t 1(· · · for ɛ small enough so that ∆(ɛ) > K)ɛ√ 8∆(ɛ)3ɛ 2 Clip2 ¯KT + 8∆(ɛ) ∫ tɛ 2 Clip2 E[||x ɛ 1(s) − x ɛ (s)|| 2 ]ds,t 1where C lip is a common Lipschitz constant for the functions involved in the definitionof the SP CSDE (1)-(2) and it has been taken into account that, by (26),max s∈[tl−1 ,t l ] E[||z ɛ (s) − z ɛ (t l )|| 2 ] ≤√¯KT ∆(ɛ) . By Gronwall inequality,√E[||x ɛ 1(t) − x ɛ (t)|| 2 ] ≤ 8∆(ɛ)3ɛ 2 Clip2 ¯KT e 8∆2 (ɛ)ɛ 2 Clip 2 ∀t ∈ [t 1 , t 2 ].Define ∆(ɛ)as follows√∆(ɛ) def = ɛ aln 1 ɛ ,a def = (32C 2 lip) −1 . (93)Note that with such a definition of ∆(ɛ) (which we are going to use from now on)the relationships (83) are satisfied and8∆(ɛ) 3 √ √ɛ 2 Clip2 ¯KT e 8∆2 (ɛ)ɛ 2 Clip 2 = 8Clip2 ¯KT a − 3 12 ɛ[lnɛ ] 3 2 eln( 1 ɛ ) 14 ≤ ɛ 1 229

for ɛ sufficiently small. The latter implies that the following inequality is valid forsufficiently small ɛmax E[t∈[t 1 ,t 2 ] ||xɛ 1(t) − x ɛ (t)|| 2 ] ≤ ɛ 1 2 . (94)Assume now that the solution (z ɛ (t), x ɛ (t), u ɛ (t)) of (1)-(2) has been definedon the interval [0, t l ) and extend the definition to the interval [0, t l+1 ). Take¯θ ldef= Ψ(z ɛ (t l ), ¯θ l ) , ¯v ldef= ψ 1 (z ɛ (t l ), ¯θ l ) , ¯µ ldef= ψ 2 (z ɛ (t l ), ¯θ l ) .Similarly to (89), one can obtain thatNote that (see (81))||¯v l − ¯v l || + ˆρ(¯µ l , ¯µ l ) = ˆβ(¯θl , ¯θ l )= min ˆβ(θ, ¯θ l )θ∈Q z ɛ (tl )≤ ˆβ H (Q z ɛ (t l ), Q z(tl ))≤ K||z ɛ (t l ) − z(t l )||. (95)¯v l = ¯h(z ɛ (t l ), ¯µ l ) (96)Consider the associated system (6) with z = z ɛ (t l ), W ′ (τ) = W (ɛτ)−W √ (t l)ɛandwith the initial condition x(0) = x ɛ (t l ). By Assumptions 2, there exists a solution(x l (·), u l (·)) of this system such thatE[ |(S ɛ ) −1 ∫ Sɛ0∫f i (x 1 (τ), u 1 (τ))dτ −for any f i (x, u) used in the definition of the metric (33), andE[ ||(S ɛ ) −1 ∫ Sɛ0∫h(z ɛ (t 1 ), x 1 (τ), u 1 (τ))dτ −f i (x, u)¯µ 1 (dx, du)| ] ≤ ν i (ɛ), (97)h(z ɛ (t 1 ), x, u)¯µ 1 (dx, du)|| ] 2≤ ν(ɛ),(98)where S ɛ , ν i (ɛ) and ν(ɛ) are as in (90) and (91). That is, in particular, S ɛ → ∞and both ν(ɛ) → 0 and ν i (ɛ) → 0 as ɛ → 0.Note that, using Cauchy-Schwartz inequality and Assumption 1, one can easilyverify that (98) impliesE[ ||(S ɛ ) −1 ∫ Sɛ0∫h(z ɛ (t l ), x ′ l(τ), u ′ l(τ))dτ −√h(z ɛ (t l ), x, u)¯µ l (dx, du)|| 2 ] ≤ K ν(ɛ).(99)Extend the definition of the solution (z ɛ (t), x ɛ (t), u ɛ (t)) of the system (1)-(2) tothe interval [0, t l+1 ] by applying the control u ɛ (t) def= u l ( t−t lɛ) on the interval t ∈[t l , t l+1 ).30

It is proved below that the solution of (1)-(2) thus constructed satisfies (42).Before passing to this proof let us define x ɛ l = x l ( t−t lɛ). Note that, replacingthe time scale t = t l + ɛτ, one can obtain (similarly to (94)) thatand also to rewrite (99),(97) in the forms1E[ ||∆(ɛ)and∫ tl+1t l1E[ |∆(ɛ)respectively.(t)defmax E[t∈[t l ,t l+1 ] ||xɛ l (t) − x ɛ (t)|| 2 ] ≤ ɛ 1 2 . (100)∫h(z ɛ (t l ), x ɛ l (t), u ɛ (t))dt −∫ tl+1Lemma 7.3 For any T > 0,t l∫f i (x ɛ l (t), u ɛ (t))dt −√h(z ɛ (t l ), x, u)¯µ l (dx, du)|| 2 ] ≤ K ν(ɛ),(101)f i (x, u)¯µ l (dx, du)| ] ≤ Kν i (ɛ) (102)lim max E[ɛ→0 t∈[0,T ] ||zɛ (t) − z(t)|| 2 ] = 0. (103)Proof. By definition, z ɛ (t) and z(t) satisfy the equations:∫ t∫ tz ɛ (t) = z 0 + h(z ɛ (t ′ ), x ɛ (t ′ ), u ɛ (t ′ ))dt ′ + γ(z ɛ (t ′ ))dB(t ′ ),00∫ t∫ tz(t) = z 0 + ¯h(z(t ′ ), µ(t ′ ))dt ′ + γ(z(t ′ ))dB(t ′ ).00Using the fact that γ(z) satisfies Lipschitz conditions, one can obtain that∫ t∫ tE[ ||z ɛ (t) − z(t)|| 2 ] ≤ K{E[ || h(z ɛ (t ′ ), x ɛ (t ′ ), u ɛ (t ′ ))dt ′ − ¯h(z(t ′ ), µ(t ′ ))dt ′ || 2 ]00∫ t+ E[ ||z ɛ (t ′ ) − z(t ′ )|| 2 ]dt ′ }. (104)0Let us evaluate the first term in the right-hand-side of (104). Let t ∈ [0, T ] anddefk t = ⌊ tt∆(ɛ)⌋, that is the integer part of∆(ɛ) . Then∫ tE[ || h(z ɛ (t ′ ), x ɛ (t ′ ), u ɛ (t ′ ))dt ′ −0∫ t031¯h(z(t ′ ), µ(t ′ ))dt ′ || 2 ]

≤ K{E[ ||≤k t −1 ∑l=1K{k t ∆(ɛ)E[+k t ∆ 2 (ɛ)E[+k t ∆ 2 (ɛ)E[+k t ∆ 2 (ɛ)E[+k t E[+k t E[∫ tl+1t lk t −1 ∑l=1k t −1 ∑l=1k t−1 ∑l=1k∑t−1l=1(k∑t−1 ∫ tl(h(z ɛ (t ′ ), x ɛ (t ′ ), u ɛ (t ′ )) − ¯h(z(t ′ ), µ(t ′ )) )dt ′ || 2 ] + ∆(ɛ)}∫ tl+1t l||h(z ɛ (t ′ ), x ɛ (t ′ ), u ɛ (t ′ )) − h(z ɛ (t l ), x ɛ l (t ′ ), u ɛ (t ′ ) )|| 2 dt ′ ]∫ tl+1||∆ −1 (ɛ) ( h(z ɛ (t l ), x ɛ l (t ′ ), u ɛ (t ′ ))dt ′ − ¯h(z ɛ (t l ), ¯µ l ) || 2 ]t l||¯h(z ɛ (t l ), ¯µ l ) − ¯h(z ɛ (t l ), ¯µ l )|| 2 ]||¯h(z ɛ (t l ), ¯µ l ) − ¯h(z ɛ (t l ), ˜µ l )|| 2 ]) 2||¯h(z ɛ (t l ), µ(t ′ )) − ¯h(z(t l ), µ(t ′ ))||dt ′ ]l=1t l−1(k∑t −1 ∫ tll=1t l−1||¯h(z(t l ), µ(t ′ )) − ¯h(z(t ′ ), µ(t ′ ))||dt ′ ) 2] + ∆(ɛ)}.Let us estimate the right-hand-side terms of the above inequality. From the Lipschitzcontinuity of h(z, x, u) in (z, x) and from (26), (100) it follows thatk t ∆(ɛ)E[≤ k t ∆(ɛ)KE[k t−1 ∑l=1k t−1 ∑l=1∫ tl+1t l||h(z ɛ (t ′ ), x ɛ (t ′ ), u ɛ (t ′ )) − h(z ɛ (t l ), x ɛ l (t ′ ), u ɛ (t ′ ) )|| 2 dt ′ ]∫ tl+1Using (101), one can show thatt l(||z ɛ (t ′ )−z ɛ (t l )|| 2 +||x ɛ (t ′ )−x ɛ l (t ′ )|| 2 )dt ′ ] ≤ K(∆(ɛ)+ɛ 1 2 ).kk t ∆ 2 ∑ t −1 ∫ tl+1√(ɛ)E[ ||∆ −1 (ɛ) ( h(z ɛ (t l ), x ɛ l (t ′ ), u ɛ (t ′ ))dt ′ −¯h(z ɛ (t l ), ¯µ l ) || 2 ] ≤ K ν(ɛ).t ll=1From the fact that h(z, x, u) satisfies Lipschitz conditions in z it follows that ¯h(z, µ)satisfies Lipschitz conditions in z. Using this as well as (84), (96), (85),(95) and(26),(87), one can verify thatk t (∆ 2 (ɛ)E[+∆ 2 (ɛ)E[k t −1 ∑l=1k∑t −1l=1||¯h(z ɛ (t l ), ¯µ l ) − ¯h(z ɛ (t l ), ¯µ l )|| 2 ]||¯h(z ɛ (t l ), ¯µ l ) − ¯h(z ɛ (t l ), ˜µ l )|| 2 ])32

≤≤≤⎛K∆(ɛ) ⎝E[¯K∆(ɛ){E[k t −1 ∑l=1k t −1 ∑l=1||¯v l − ¯v l || 2 ] + E[k t −1 ∑l=1||z ɛ (t l ) − z(t l )|| 2 ] + 1}∫ t¯K{ E[ ||z ɛ (t ′ ) − z(t ′ )|| ] 2 dt ′ + ∆(ɛ)},0||¯v l − ṽ l || 2 ] + E[k t −1 ∑l=1⎞||z ɛ (t l ) − z(t l )|| 2 ] + 1⎠where K, ¯K, ¯K are positive constants. Again, using the Lipschitz continuity of¯h(z, µ) in z and (26), (87), one can verify thatand thatk t E[k t E[≤ K{k t −1 ∑l=1k t−1 ∑l=1∫ t0( ∫ tl( ∫ tlt l−1||¯h(z ɛ (t l ), µ(t ′ )) − ¯h(z(t l ), µ(t ′ ))||dt ′ ) 2]E[ ||z ɛ (t ′ ) − z(t ′ )|| 2 ]dt ′ + ∆(ɛ)},t l−1||¯h(z(t l ), µ(t ′ )) − ¯h(z(t ′ ), µ(t ′ ))||dt ′ ) 2] ≤ K∆(ɛ).Summarizing the above estimates in (104) and applying Gronwall-Bellman Lemma,one can obtain now thatwhich proves (103).E[ ||z ɛ (t) − z(t)|| 2 ] ≤ K(∆(ɛ) + ɛ 1 2 +√ν(ɛ) ) ∀t ∈ [0, T ], (105)Lemma 7.4 Let µ ɛ (t, dxdu) def= δ (x ɛ (t),u ɛ (t))(dxdu). Then, for any T > 0 ,Proof. By (85), (95) and (105),lim E[g T (µ ɛ (·), µ(·))] = 0, (106)ɛ→0E[ ˆρ(¯µ l , ˜µ l ) ] ≤ ˜ν(ɛ) ⇒ E[ ρ(¯µ l , ˜µ l ) ] ≤ ˜ν(ɛ) , (107)where ˜ν(ɛ) def= K(∆(ɛ) + ɛ 1 2 + ν 1 2 (ɛ)) 1 2 and ρ(·, ·) is defined in (33). Hence,E[ |〈f i , ¯µ l 〉 − 〈f i , ˜µ l 〉| ] ≤ (2c i ) i˜ν(ɛ) . (108)Let αi ɛ def(t) = 〈f i , µ ɛ (t)〉 and α i (t) def= 〈f i , µ(t)〉 . From the definitions ofµ ɛ (t) and ˜µ l it follows that✷∫ tl+1t lα ɛ i(t)dt =∫ tl+1t lf i (x ɛ (t), u ɛ (t))dt,∫ tlt l−1α i (t)dt = ∆(ɛ)〈f i , ˜µ l 〉 .33

Hence, for any 0 < ¯t < ¯t < T ,≤=≤∫ ¯t ∫E[ | αi(t)dt ɛ ¯t− α i (t)dt| ]¯t¯tN ɛ −1 ∑l=1N ɛ −1 ∑E[ |l=1N∑ɛ−1 ∫ tl+1l=1∫ tl+1t l∫ tlE[ |t ll=1α ɛ i(t)dt −N∑ɛ−11+∆(ɛ) E[ |∆(ɛ)∫ tlt l−1α i (t)dt| ] + O(∆(ɛ))t l−1f i (x ɛ (t), u ɛ (t))dt − ∆(ɛ)〈f i , ˜µ l 〉| ] + O(∆(ɛ))E[ |f i (x ɛ (t), u ɛ (t)) − f i (x ɛ l (t), u ɛ (t))| ]dt∫ tl+1t lf i (x ɛ l (t), u ɛ (t))dt − 〈f i , ¯µ l 〉| ]N∑ɛ −1+∆(ɛ) E[ |〈f i , ¯µ l 〉 − 〈f i , ˜µ l 〉| ] + O(∆(ɛ)),l=1where N ɛ = ⌊ T∆(ɛ) ⌋. By (100) and the Lipschitz continuity of f i,By (102),N ɛ −1 ∑l=1N∑ɛ−1∆(ɛ)l=1Finally, by (108),∫ tl+1t lE[ |f i (x ɛ (t), u ɛ (t)) − f i (x ɛ l (t), u ɛ (t))| ]dt ≤ Kɛ 1 4 .1E[ |∆(ɛ)N∑ɛ−1∆(ɛ)l=1Thus, for any 0 < ¯t < ¯t < T ,∫ tl+1t lf i (x ɛ l (t), u ɛ (t))dt − 〈f i , ¯µ l 〉| ] ≤ Kν i (ɛ) .E[ |〈f i , ¯µ l 〉 − 〈f i , ˜µ l 〉| ] ≤ K ˜ν(ɛ) .∫ ¯t ∫E[ | αi(t)dt ɛ ¯t− α i (t)dt| ] ≤ O(˜ν(ɛ)) + O(ν i (ɛ)) → 0¯t¯tas ɛ → 0. This implies that lim ɛ→0 E[ḡ T (αi ɛ(·), α i(·))] = 0 , which, in turn, impliesthe validity of (106).✷34

Proof of Proposition 4.4. Let µ be an extreme point of D z . By Theorem 2.1 in [15]or Theorem 4.1 in [59] (see also [47]), there exists a stationary solution (x(·), v(·))of the relaxed associated system (49) such that its one dimensional marginal lawis equal to µ. That is,∫∫E[ f i (x(t), u)v(t)(du)] = f i (x, u)µ(dxdu), i = 1, 2, ... , (109)A¯R s ×Awhere f i (·) are as in (33). To prove (51), it is enough to show that1limS→∞ S∫ S ∫0A∫f i (x(t), u)v(t)(du) =f i (x, u)µ(dxdu) a. s. , i = 1, 2, ... .¯R s ×A(110)The pair (x(·), v(·)) can be viewed as a random variable taking values inC((−∞, ∞); R s ) × V, where C((−∞, ∞); R s ) is the space of continuous functionsfrom (−∞, ∞) to R s with the topology of uniform convergence on compactsubsets and V def= the space of measurable maps from (−∞, ∞) to P(A)with the coarsest topology that renders continuous the maps ψ f,g,t,θ (v ′ (·)) def=∫ θt g(τ) ∫ A f(u)v′ (τ)(du)dτ → R ∀t < θ, ∀g ∈ L 2 [t, θ], ∀f ∈ C(A).Denote by ˜W z the set of laws of stationary solutions of the relaxed associatedsystem (49) viewing it as a subset of P(C((−∞, ∞); R s ) × V) (the space of probabilitymeasures defined on Borel subsets of C((−∞, ∞); R s ) × V). It is easy tosee that ˜W z is convex and it can be verified (using Assumption 1 ′ and argumentssimilar to those used in Proposition 3.1 and Corollary 3.2) that it is relativelycompact in P(C((−∞, ∞); R s ) × V). Also, it can be shown that ˜W z is closed and,thus, compact.Let ˜W z ∗ ⊂ ˜W z be a set of laws of relaxed stationary solutions that have µ astheir one dimensional marginal law (that is, (109) is satisfied for every element of˜W z ∗ ). The set ˜W z ∗ is convex and compact, and it can be shown that every extremepoint of ˜W∗ z is also an extreme point of ˜Wz . In fact, if L(x ∗ (·), v ∗ (·)) is an extremepoint of ˜W∗ z that is not an extreme point of ˜Wz , then L(x ∗ (·), v ∗ (·)) is a convexcombination of two distinct L(x 1 (·), v 1 (·)) ∈ ˜W z and L(x 2 (·), v 2 (·)) ∈ ˜W z , and,hence, µ is a convex combination of distinct elements µ 1 ∈ D z and µ 2 ∈ D z , beingone dimensional marginal laws of L(x 1 (·), v 1 (·)) and L(x 2 (·), v 2 (·)) respectively.This would contradict to the assumption that µ is an extreme point of D z .Thus, without loss of generality, one may assume that the law L(x(·), v(·)) ofthe solution (x(·), v(·)), which satisfies (109) is an extreme point of ˜Wz . To prove(110), it is sufficient to prove that this law is ergodic.The Borel σ−field of C((−∞, ∞); R s ) × V is generated by cylinder sets of thetypeB = {(x ′ (·), v ′ (·)) : x ′ (t i ) ∈ B i , 1 ≤ i ≤ m;35∫ s ′js j∫g j (τ) f j (u)v ′ (τ)(du)dτ ∈ C j ,A

1 ≤ j ≤ n},where for some n, m ≥ 1, t 1 < t 2 < · · · < t m , s 1 < s 2 < · · · < s n , and s i < s ′ iin R, f j (·) ∈ C(A), g j (·) ∈ L 2 [s j , s ′ j ], 1 ≤ j ≤ n, and B i’s, C j ’s are Borel in therespective spaces. Call such a cylinder B a “pre-t set” if t m ≤ t and max j s ′ j ≤ t.Define F t ′ = σ(x(s), v(s), s ≤ t), F −∞ ′ = ∩F t, ′ F ∞ ′ = ∨ t F t ′ and F I ′ = {B ∈ F ∞ ′ : Bis shift invariant}, all completed w.r.t. the underlying probability measure. ThusF t ′ is generated by the pre-t sets. We claim that F I ′ ⊂ F −∞.′To see this, first note that the collection of G ∈ F ∞ ′ such that for any δ > 0there exists a pre-t set B with P (B∆G) < δ, coincides with F t. ′ Also, denote byθ t the t−shift that maps (x ′ (·), v ′ (·)) ∈ C((−∞, ∞); R s ) × V to (x ′ (t + ·), v ′ (t + ·)).Take arbitrary D ∈ F I ′ . For any δ > 0, there exists a pre-t set B withP (B∆D) < δ. Fix τ ∈ R and letT > max{(t m − τ) + , (s ′ j − τ) + , 1 ≤ j ≤ n}.Then θ −T (B) is a pre-τ set. By shift-invariance of D and stationarity of (x(·), v(·)),P (θ −T (B)∆D) = P (θ −T (B)∆θ −T (D))= P (B∆D) < δ.Hence D ∈ F τ ′ and, as τ is arbitrary, D ∈ F −∞. ′ Consequently, F I ′ ⊂ F −∞.′Let us now prove that L(x(·), v(·)) is ergodic. Assume that it is not. Then thereexists some D ∈ F I ′ with P (D) ∈ (0, 1). Then L(x(·), v(·)) = P (D)L(x(·), v(·)|D)+(1 − P (D)L(x(·), v(·)|D c ), with L(x(·), v(·)|D) and L(x(·), v(·)|D c ) being the lawsof (x(·), v(·)) conditioned on D and D c resp. Now the membership of L(x(·), v(·))in ˜W z is characterized by the fact that∫ θ ∫f(x(θ)) − Lf(z, x(τ), u)v(τ)(du)dτ, θ ≥ t,tAis a martingale w.r.t. F t ′ for any f(·) ∈ C0 2(Rs ). That is,∫ θ ∫E[(f(x(θ)) − f(x(t)) − Lf(z, x(τ), u)v(τ)(du)dτ)|F t] ′ = 0 a. s. (111)tAfor all θ > t. Since D ∈ F I ′ ⊂ F −∞, ′ the aforementioned martingale propertyimplies that∫ θ ∫E[(f(x(θ)) − f(x(t)) − Lf(z, x(τ), u)v(τ)(du)dτ)I D |F t] ′ = 0 a. s.tADividing by P (D) > 0, we get (111) under L(x(·), v(·)|D). A similar argumentworks for L(x(·), v(·)|D c ). Thus L(x(·), v(·)|D) ∈ ˜W z and L(x(·), v(·)|D c ) ∈ ˜W z ,which contradicts the extreme point property of L(x(·), v(·)). Hence L(x(·), v(·))must be ergodic.✷36

8 Proofs of Propositions 5.2, 5.3 and of Lemma2.2(ii)Proof of Proposition 5.2. The validity of Assumption 2 has been already established.To prove the validity of Assumption 3, take an arbitrary (v, µ) ∈ Q z 1.That is, µ ∈ D z 1 and v = ¯h(z 1 , µ). From Assumption 2 it follows that there existan admissible control u(τ) and the corresponding solution x z 1(τ) of the associatedsystem (considered with z = z 1 ) such thatlim E[ | 1 ∫ S∫f i (xS→∞ Sz 1(τ), u(τ))dτ −0f i (x, u)µ(dx, du)| ] = 0 , (112)where f i (·) are as in (33). Also, from the fact that the sequence of f i (·), i = 1, 2, ...,is dense in the unit ball of C( ¯R s × A) it follows thatlim E[ || 1 ∫ S∫h(z 1 , xS→∞ Sz 1(τ), u(τ))dτ −0h(z 1 , x, u)µ(dx, du)|| ] = 0 . (113)Denote by x z 2(τ) the solution of (6) obtained with the same control, same initialconditions, same Brownian motion (as those x z 1(τ) was obtained with) but withz = z 2 . Similarly to Lema 4.1 in [33], it can be established thatE[ ||x z 1(τ) − x z 2(τ)|| ] ≤ c ′ ||z 1 − z 2 || ∀τ ≥ 0.where c ′ = const. Hence, from the Lipschitz continuity of f i (x, u) (with a constantc i ) and from the Lipschitz continuity of h(z, x, u) in (z, x) (with a constant c lip ) itfollows that, for any S > 0 ,E[ | 1 Sand∫ S0f i (x z 1(τ), u(τ))dτ − 1 S∫ S∫ S0f i (x z 2(τ), u(τ))dτ| ] ≤ c i c ′ ||z 1 − z 2 || , (114)E[ || 1 h(z 1 , xSz 1(τ), u(τ))dτ − 1 h(z 2 , x0Sz 2(τ), u(τ))dτ|| ]0≤ c lip (1 + c ′ )||z 1 − z 2 || . (115)Let µ S be the occupational measure of the process (x z 2(τ), u(τ)) on the interval[0, S] and let v Sdef= ¯h(z 2 , µ S ). From (38) and the triangle inequality it follows thatβ((v, µ), Q z 2) ≤ lim sup E[ ||v − v S || + ρ(µ, µ S ) ] + lim sup E[ β((v S , µ S ), Q z 2) ] .S→∞S→∞(116)∫ S37

Note that, by Proposition 5.1,lim sup E[ β((v S , µ S ), Q z 2) ] = 0 . (117)S→∞Taking into account the definition of the occupational measure as well as (113)and (115), one can obtain thatlim sup E[ ||v − v S || ]S→∞∫= lim sup E[ || h(z 1 , x, u)µ(dx, du) − 1S→∞S∫≤ lim sup E[ || h(z 1 , x, u)µ(dx, du) − 1S→∞S∫ S∫ S0∫ S+ lim sup E[ || 1 h(z 1 , xS→∞ Sz 1(τ), u(τ))dτ − 10S≤ c lip (1 + c ′ )||z 1 − z 2 || .0h(z 2 , x z 2(τ), u(τ))dτ|| ]h(z 1 , x z 1(τ), u(τ))dτ|| ]Similarly, taking into account (112) and (114), one obtains thatlim sup E[|〈f i , µ〉 − 〈f i , µ S 〉|]=S→∞∫lim sup E[|S→∞f i (x, u)µ(dx, du) − 1 S≤ c i c ′ ||z 1 − z 2 ||∫ S⇒ lim sup E[ ρ(µ, µ S ) ]S→∞∞∑= lim sup E[ (2c i ) −i |〈f i , µ〉 − 〈f i , µ S 〉| ]S→∞≤ c ′ ||z 1 − z 2 ||.i=10∫ S0h(z 2 , x z 2(τ), u(τ))dτ|| ]f i (x z 2(τ), u(τ))dτ| ]The above inequalities along with (117) imply (being substituted into (116)) thatβ((v, µ), Q z 2) ≤ (c lip (1 + c ′ ) + c ′ )||z 1 − z 2 || .Since (v, µ) is an arbitrary element of Q z 1 and since z 1 and z 2 are symmetrical,this implies the validity of (40) with c ∗ = c lip (1 + c ′ ) + c ′ .✷Proof of Proposition 5.3. We denote the right hand side of (63) by P z and showthat Ω z = P z .Let ū be an arbitrary element of A and ¯x(τ) be a solution of the associatedsystem (6) obtained with the control ū(τ) def= ū. Let also ¯µ S be the occupational38

measure generated by the pair (¯x(·), ū(·)) on the interval [0, S] and let∫¯x S = x¯µ S (dxdu) = 1R s ×A Sthe last equality being implied by (35). By (6),∫ S0¯x(τ)dτ ,¯x(S) − ¯x(0)S= F 2 (z)¯x s + H 2 (z)ū + 1 S∫ S0σ(z)dW ′ (τ) .Since E[ || ¯x(S)−¯x(0)S|| 2 ] → 0 and E[ 1 S || ∫ S0 σ(z)dW ′ (τ)|| 2 ] → 0 as S → ∞, one canconclude that E[ ||F 2 (z)¯x s + H 2 (z)ū|| 2 ] → 0 as S → ∞. Hence,lim E[ ||¯x s − (−F2 −1 (z)H 2 (z)ū )|| ] = 0 . (118)S→∞Let q(x, u) = (x, u) . Then Vz q = Ω z , where VzqProposition 5.1. By (54),is the set introduced in∫lim E[ dist( q(¯x s, ū) , Ω z ) ] = lim E[ dist( q(x, u)¯µ S (dxdu) , Ω z ) ] = 0S→0 S→0 R s ×Aand, by (118),lim E[ ||q(¯x s, ū) − (−F2 −1 (z)H 2 (z)ū , ū )|| ] = 0 .S→∞From the fact that Ω z is closed it follows now that (−F −12 (z)H 2 (z)ū , ū ) ∈ Ω zand, consequently, P z ⊂ Ω z (since ū is an arbitrary element of A). To provethe converse inclusion, denote: f 1,N (x) = x 1 φ N (x) , ... , f s,N (x) = x s φ N (x) ,where x = (x 1 ... x s ) and φ N (x) : R s → R 1 is a function having continuous firstand second derivatives such that φ N (x) = 1 for ||x|| ≤ N and φ N (x) = 0 for||x|| ≥ N + 1. It is easy to see thatLf l,N = m l (z, x, u) for ||x|| ≤ N , Lf l,N = 0 for ||x|| ≥ N + 1and |Lf l,N (z, x, u)| is bounded for N < ||x|| < N + 1 , where m l (·) are the componentsof m(·). From (56) and the Lipschitz continuity of m l (z, x, u) in x it followsthat, for any µ ∈ D z and N → ∞,∫∫0 = Lf l,N (z, x, u)µ(dxdu) → m l (z, x, u)µ(dxdu)R s ×A⇒∫R s ×AR s ×Am(z, x, u)µ(dxdu) = 0 ∀µ ∈ D z . (119)39

Let (¯x, ū) be an arbitrary element of Ω z . That is, (¯x, ū) = ∫ R s ×A(x, u)µ(dxdu) forsome µ ∈ D z . Taking into account (60) and (119), one obtains¯x = −F −12 (z)H 2 (z)ū ⇒ (¯x, ū) ∈ P z ,where it is taken into account that ū ∈ A (since A is convex). The last inclusionimplies that Ω z ⊂ P z .✷Proof of Lemma 2.2 (ii) We shall prove the claim for U, the other case beingsimilar. Let µ n (·) ∈ U, n = 1, 2, ... and let αi n def(t) = ∫ R s ×A f i(x, u)µ n (t)(dxdu),where the sequence f i (·), i = 1, 2, ..., is the same as in (33) (that is, it is densein the unit ball of C( ¯R s × A)). Due to the fact that the unit ball of L 2 [0, T ] isweakly compact for any T > 0, one may assume (by dropping to a subsequence ifnecessary) that there exist measurable functions α i (·) : [0, ∞) → R 1 , i = 1, 2, ...such that, for every i, αi n(·) converges weakly to a α i(·) on any interval [0, T ].Fix T > 0. Let n(1) = 1 and define {n(k)} inductively so that∞∑i=1∫ T2 −i max 1≤l

This is well-defined because {f i (·)} is dense in the unit ball of C( ¯R s × A). In fact,ν k (t) = 1m(k)∑µ n(j) (t), i ≥ 1, k ≥ 1.m(k)j=1For every t ∈ [0, ∞), define µ(t) as a limit point of {ν k (t)} in P( ¯R s ×A) as k → ∞.By (120) and (121),∫f i (x, u)µ(t)(dxdu) = α i (t), i ≥ 1 (122)R s ×Afor a.e. t, where the qualification “a.e.” can be dropped by suitably modifying µ(·)on a Lebesgue-null set. Hence, µ(·) ∈ U.From the weak convergence of α n i (·), i = 1, 2, , ..., to the corresponding α i(·), i =1, 2, , ..., on any interval [0, T ] it follows thatlimn→∞ g(µn (·), µ(·)) = 0 , (123)where g(·, ·) is defined in (21). This proves the required statement.✷References[1] Altman E., Gaitsgory V. Asymptotic Optimization of a Nonlinear HybridSystem Governed by a Markov Decision Process, SIAM J. Control and Optimization,35 (1997), no. 6, pp.2070-2085.[2] Alvarez O., Bardi M., Viscosity Solutions Methods for Singular Perturbationsin Deterministic and Stochastic Control, SIAM J. Control and Optimization,40(2001), pp 1159-1188.[3] Alvarez O., Bardi M., Singular Perturbations of Nonlinear DegenerateParabolic PDEs: a General Convergence result, Arch. Ration. Mech. Anal.,170 (2003), pp. 17-61.[4] Artstein Z., Invariant Measures of Differentialinclusions Applied to SingularPerturbations, Journal of Differential Equations, 152 (1999), pp 289-307.[5] Artstein Z., An Occupational Measure Solution to a Singularly Perturbed OptimalControl Problem, Control and Cybernetics, 31 (2002), pp. 623-642.[6] Artstein Z. , “Invariant Measures and Their Projections in NonautonomousDynamical Systems”, Stochastics and Dynamics, 4 (2004), no. 3, pp 439-459.41

[7] Artstein Z., Gaitsgory V., Tracking Fast Trajectories Along a Slow Dynamics:A Singular Perturbations Approach, SIAM J. Control and Optimization, 35(1997), no 5, pp. 1487-1507.[8] Artstein Z., Gaitsgory V., The Value Function of Singularly Perturbed ControlSystems, Appl. Math. Optim., 41 (2000), pp 425-445.[9] Artstein Z. and Leizarowitz A., “Singularly Perturbed Control Systemswith One-Dimensional Fast Dynamics”, SIAM J. Control and Optimization,41(2002), 641-658.[10] Artstein Z. and Vigodner A., “Singularly Perturbed Ordinary Differentialequations with Dynamic Limits”, Procceding of the Royal Society of Edinburgh,126A (1996), 541-569.[11] Avrachenkov K.E., Filar J. and Haviv M. Singular Perturbations of MarkovChains and Decision Processes, in “Handbook of Markov Decision Processes”.Methods and Applications, Kluwer, Boston, 2002.[12] Bagagiolo F., Bardi M., Singular Perturbation of a Finite Horizon Problemwith State-Space Constraints, SIAM J. Control and Optimization, 36 (1998),pp. 2040-2060.[13] Basak, G. K., Borkar, V. S., Ghosh, M. K., Ergodic Control of DegenerateDiffusions, Stochastic Analysis and Appl. 15 (1997), 1-17.[14] Bensoussan A., Perturbation Methods in Optimal Control, John Wiley, NewYork, 1989.[15] Bhatt A. G., Borkar V. S., Occupation Measures for Controlled Markov Processes:Characterization and Optimality, Annals of Probability 24 (1996),1531-1562.[16] Billingsley P., Convergence of Probability Measures, John Wiley, New York,1968.[17] Borkar V. S., Optimal Control of Diffusion Processes, Pitman Research Notesin Maths. No. 203, Longman Scientific and Technical, Harlow, UK, 1989.[18] Borkar V. S., Probability Theory: An Advanced Course, Springer Verlag, NewYork, 1995.[19] Borkar V. S., Stability of Annealing Schemes and Related Processes, Systemsand Control Letters 41 (2000), 325-331.42

[20] Borkar V. S., Gaitsgory V. On Existence of Limit Occupational MeasuresSet of a Controlled Stochastic Differential Equation, SIAM J. Control andOptimization, 44 (2005), pp 1436-1473.[21] Colonius F. and Fabbri R, “Controllability for Systems with Slowly VaryingParameters”, ESAIM: Control, Optimisation and Calculus of Variations,9(2003), pp 207-216.[22] Donchev T. and Slavov I., Averaging Method for One Sided Lipschitz DifferentialInclusions, SIAM J. Control and Optimization, 37 (1999), pp 1600-1613.[23] Donchev T.D. and Dontchev A.L., Singular Perturbations in Infinite-Dimensional Control Systems, SIAM J. Control and Optimization, 42 (2003),pp. 1795-1812.[24] Dontchev A., Time-scale Decomposition of the Reachable Set of ConstrainedLinear Systems, Mathematics of Control, Signal, and Systems, 5 (1992),pp. 327-340.[25] Dontchev A., Donchev T. and Slavov I, A Tichonov-type Theorem for SingularlyPerturbed Differential Inclusion, Nonlinear Analysis, 26 (1996), 1547-1554.[26] Dudley R. M., Distances of Probability Measures and Random Variables, TheAnnals of Mathematical Statistics 39 (1968), 1563-1572.[27] Dudley R.M., Real Analysis and Probability, Wadsworth and Brooks/ColeMathematics Series, Pacific Grove, California, 1989.[28] Dunford N., Schwartz J.T., Linear Operators. Part I: General Theory, IntersciencePublishers, New York, 1958.[29] Ethier, S., T.G. Kurtz, Markov Processes: Characterization and Convergence,John Wiley, New York, 1986.[30] Filar J.A., V. Gaitsgory V. and Haurie A., “Control of Singularly PerturbedHybrid Stochastic Systems”, IEEE Trans. on Automatic Control, 46:2 (2001).[31] Filatov O.P., Hapaev M.M., Averaging of Systems of Differential Inclusions,Moscow University Publishing House, Moscow, 1998 (in Russian).[32] Folland, G.B., Real Analysis, John Wiley, New York, 1984.[33] Gaitsgory V., Suboptimization of Singularly Perturbed Control Problem, SIAMJ. Control and Optimization, 30 (1992), No. 5, pp. 1228 - 1248.43

[34] Gaitsgory V., On a Representation of the Limit Occupational Measures Setwith Applications to Singularly Perturbed Control Systems, SIAM J. Controland Optimization, 43 (2004), pp. 325-340.[35] Gaitsgory V., Leizarowitz A., Limit Occupational Measures Set for a ControlSystem and Averaging of Singularly Perturbed Control Systems, J. Math.Anal. and Appl., 233 (1999), pp. 461-475.[36] Gaitsgory V., Nguyen M.T., Multiscale Singularly Perturbed Control Systems:Limit Occupational Measures Sets and Averaging, SIAM J. Control Optim.,41(2002), No. 3, pp. 954-974.[37] Gilbarg D., Trudinger N. S., Elliptic Partial Differential Equations of SecondOrder, Springer Verlag, Berlin-Heidelberg, 2001.[38] Grammel G., Averaging of Singularly Perturbed Systems, Nonlinear Analysis,28 (1997), pp. 1855-1865.[39] Grammel G., Exponential Stability of Nonlinear Singularly perturbed DifferentialEquations , SIAM J. Control and Optimization, 44 (2005), pp. 1712-1724.[40] Kabanov Y., Pergamenshikov S., On Convergence of Attainability Sets forControlled Two-Scale Stochastic Linear Systems, SIAM J. on Control andOptimization, 35 (1997), pp. 134-159.[41] Kabanov Y., Pergamenshchikov S., Two-Scale Stochastic Systems, SpringerVerlag, Berlin-Heidelberg, 2003.[42] Khasminskii R.Z., On the Averaging Principle for Ito Stochastic Equations,Kybernetika, 4:3 (1968), pp. 260-279[43] Khasminskii R.Z., Stochastic Stability of Differential Equations, Sijthoff andNoordhoff, Alphen aan den Rijn, The Netherland, 1980.[44] Khasminskii R.Z. and Yin G., On Averaging Principles: An Asymptotic ExpansionApproach, SIAM J. on Mathematical Analysis, 35:6 (2004), pp. 1534-1560[45] Kokotovic P.V., Applications of Singular Perturbation Techniques to ControlProblems, SIAM Review, 26 (1984), 501-550.[46] Kokotovic P.V., Khalil H.K., and O’Reilly J., Singular Perturbation Methodsin Control: Analysis and Design, Academic Press, New York, 1986.44

[47] Kurtz T.G. and Stockbridge R.H., Existence of Markov Controls and Characterizationof Optimal Markov Controls, SIAM J. on Control and Optimization,36:2 (1998), pp. 609-653.[48] Kushner H.J., Weak Convergence Methods and Singularly Perturbed StochasticControl and Filtering Problems, Birkhauser, Boston, 1990.[49] Leizarowitz A., Order Reduction is Invalid for Singularly Perturbed ControlProblems with Vector Fast Variables, Math. Control Signals and Systems, 15(2002), 101-119.[50] Liptser R. S., Shiryayev A. N., Statistics of Random Processes I: GeneralTheory, Springer Verlag, New York, 1977.[51] Meyn S. P., Tweedie R. L. Markov Chains and Stochastic Stability, Springer-Verlag, London, 1993[52] O’Malley R.E., Jr., Introduction to Singular Perturbations, Academic Press,New York, 1974.[53] O’Malley R.E., Jr., Singular Perturbations and Optimal Control, In MathematicalControl Theory, W.A. Copel, ed., Lecture Notes in Mathematics, 680,Springer-Verlag, Berlin 1978.[54] Naidu S.D., Singular Perturbations and Time Scales in Control Theory andApplications: An Overview, Dynamics of Continuous Discrete and ImpulsiveSystems, Series B: Applications and lgorithms, 9 (2002), 233-278.[55] Pervozvanskii A.A., Gaitsgory V. Theory of Suboptimal Decisions, Kluwer,Dordrecht, 1988.[56] Plotnikov V.A., Plotnikov A.V., Vituk A.N. Differential Equations with MultivaluedRight-Hand Sides: Asymptotic Methods, AstroPrint, Odessa, 1999 (inRussian).[57] Quincampoix M., Zhang H., Singular Perturbations in Non-linear OptimalControl Systems, Differential and Integral Equations, 8 (1995), pp 931-944.[58] Quincampoix M.& Watbled F. Averaging method for discontinuous Mayer’sproblem of singularly perturbed control systems, Nonlinear Analysis: Theory,Methods & Applications, 54(2003), pp 819-837.[59] Stockbridge R. H., Time-average Control of a Martingale Problem: Existenceof a Stationary Solution, Annals of Probability 18 (1990), 190-205.45

[60] Stroock D. W., Varadhan S. R. S., Multidimensional Diffusion Processes,Springer Verlag, New York, 1979.[61] Vasil’eva A.B., Butuzov V.F., Asymptotic Expansions of Solutions of SingularlyPerturbed Equations, Nauka, Moscow, 1973 (in Russian).[62] Veliov V., A Generalization of Tichonov Theorem for Singularly PerturbedDifferential Inclusions, Journal of Dynamical and Control Systems, 3 (1997),pp 1-28[63] Vigodner A., Limits of Singularly Perturbed Control Problems with StatisticalLimits of Fast Motions, SIAM J. Control and Optimization, 35 (1997), 1-28.[64] Wagner D., Survey of Measurable Selection Theorems, SIAM J. Control andOptimization, 15 (1977), 859-903.[65] Watbled F., On Singular Perturbations for Differential Inclusions on the InfiniteInterval, J. Math. Anal. Appl., 310 (2005), pp.362-378.[66] Wong E., Representation of martingales, quadratic variation and applications,SIAM J. Control and Optimization, 9 (1971), 621-633.[67] Yin G.G. and Zhang Q., Continuous-Time Markov Chains and Applications.A Singular Perturbation Approach, Springer, New York, 1997.46