Lecture Notes - Department of Mathematics and Statistics - Queen's ...

More documents

Recommendations

Info

iv CONTENTS 3.2 Stability and Invariant Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.2.1 Invariant Measures via an Occupational Characterization . . . . . . . . . . . . . . . . . . 18 3.3 Uncountable (Complete, Separable, Metric) State Spaces . . . . . . . . . . . . . . . . . . . . . . . 23 3.3.1 Chapman-Kolmogorov Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.3.2 Invariant Distributions for Uncountable Spaces . . . . . . . . . . . . . . . . . . . . . . . . 25 3.3.3 Existence of an Invariant Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.3.4 Dobrushin’s Ergodic Coefficient for Uncountable State Space Chains . . . . . . . . . . . . 27 3.3.5 Ergodic Theorem for Uncountable State Space Chains . . . . . . . . . . . . . . . . . . . . 27 3.4 Further Results on the Existence of an Invariant Probability Measure . . . . . . . . . . . . . . . 27 3.4.1 Case with the Feller condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.4.2 Case without the Feller condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4 Martingales and Foster-Lyapunov Criteria for Stabilization of Markov Chains 31 4.1 Martingales . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.1.1 More on Expectations and Conditional Probability . . . . . . . . . . . . . . . . . . . . . . 31 4.1.2 Some Properties of Conditional Expectation: . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.1.3 Discrete-Time Martingales . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4.1.4 Doob’s Optional Sampling Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4.1.5 An Important Martingale Convergence Theorem . . . . . . . . . . . . . . . . . . . . . . . 34 4.1.6 Proof of the Birkhoff Individual Ergodic Theorem . . . . . . . . . . . . . . . . . . . . . . 36 4.1.7 This section is optional: Further Martingale Theorems . . . . . . . . . . . . . . . . . . . . 36 4.1.8 Azuma-Hoeffding Inequality for Martingales with Bounded Increments . . . . . . . . . . . 37 4.2 Stability of Markov Chains: Foster-Lyapunov Techniques . . . . . . . . . . . . . . . . . . . . . . 37 4.2.1 Criterion for Positive Harris Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.2.2 Criterion for Finite Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.2.3 Criterion for Recurrence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 4.2.4 On small and petite sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 4.2.5 Criterion for Transience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 4.2.6 State Dependent Drift Criteria: Deterministic and Random-Time . . . . . . . . . . . . . . 42 4.3 Convergence Rates to Equilibrium . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 4.3.1 Lyapunov conditions: Geometric ergodicity . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.3.2 Subgeometric ergodicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 5 Dynamic Programming 49
CONTENTS v 5.1 Bellman’s Principle of Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 5.2 Discussion: Why are Markov Policies Important? Why are optimal policies deterministic for such settings? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 5.3 Existence of Minimizing Selectors and Measurability . . . . . . . . . . . . . . . . . . . . . . . . . 53 5.4 Infinite Horizon Optimal Control Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.4.1 Discounted Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.5 Linear Quadratic Gaussian Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 5.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 6 Partially Observed Markov Decision Processes 63 6.1 Enlargement of the State-Space and the Construction of a Controlled Markov Chain . . . . . . . 63 6.2 Linear Quadratic Gaussian Case and Kalman Filtering . . . . . . . . . . . . . . . . . . . . . . . . 64 6.2.1 Separation of Estimation and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 6.3 Estimation and Kalman Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 6.3.1 Optimal Control of Partially Observed LQG Systems . . . . . . . . . . . . . . . . . . . . . 68 6.4 Partially Observed Markov Decision Processes in General Spaces . . . . . . . . . . . . . . . . . . 68 6.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 7 The Average Cost Problem 71 7.1 Average Cost and the Average Cost Optimality Equation (ACOE) . . . . . . . . . . . . . . . . . 71 7.2 The Discounted Cost Approach to the Average Cost Problem . . . . . . . . . . . . . . . . . . . . 72 7.2.1 Borel State and Action Spaces [Optional] . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 7.3 Linear Programming Approach to Average Cost Markov Decision Problems . . . . . . . . . . . . 73 7.4 Discussion for More General State Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 7.4.1 Extreme Points and the Optimality of Deterministic Policies . . . . . . . . . . . . . . . . 78 7.4.2 Sample-Path Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 7.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 8 Team Decision Theory and Information Structures 81 8.1 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 A On the Convergence of Random Variables 85 A.1 Limit Events and Continuity of Probability Measures . . . . . . . . . . . . . . . . . . . . . . . . . 85 A.2 Borel-Cantelli Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 A.3 Convergence of Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 A.3.1 Convergence almost surely (with probability 1) . . . . . . . . . . . . . . . . . . . . . . . . 86 A.3.2 Convergence in Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 A.3.3 Convergence in Mean-square . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Page 1 and 2: i Queen’s University Mathematics
Page 3: Contents 1 Review of Probability 1
Page 7 and 8: Chapter 1 Review of Probability 1.1
Page 9 and 10: 1.2. MEASURABLE SPACE 3 1.2.3 Measu
Page 11 and 12: 1.3. PROBABILITY SPACE AND RANDOM V
Page 13 and 14: 1.5. EXERCISES 7 where w t is an in
Page 15 and 16: Chapter 2 Controlled Markov Chains
Page 17 and 18: 2.3. PERFORMANCE OF POLICIES 11 The
Page 19 and 20: 2.4. EXERCISES 13 2.4 Exercises Exe
Page 21 and 22: Chapter 3 Classification of Markov
Page 23 and 24: 3.1. COUNTABLE STATE SPACE MARKOV C
Page 25 and 26: 3.2. STABILITY AND INVARIANT DISTRI
Page 27 and 28: 3.2. STABILITY AND INVARIANT DISTRI
Page 29 and 30: 3.3. UNCOUNTABLE (COMPLETE, SEPARAB
Page 31 and 32: 3.3. UNCOUNTABLE (COMPLETE, SEPARAB
Page 33 and 34: 3.4. FURTHER RESULTS ON THE EXISTEN
Page 35 and 36: 3.5. EXERCISES 29 3.5 Exercises Exe
Page 37 and 38: Chapter 4 Martingales and Foster-Ly
Page 39 and 40: 4.1. MARTINGALES 33 Theorem 4.1.3 I
Page 41 and 42: 4.1. MARTINGALES 35 Theorem 4.1.6 L
Page 43 and 44: 4.2. STABILITY OF MARKOV CHAINS: FO
Page 49 and 50: 4.3. CONVERGENCE RATES TO EQUILIBRI
Page 51 and 52: 4.4. CONCLUSION 45 The second condi
Page 53 and 54: 4.5. EXERCISES 47 ∀ bounded funct
Page 55 and 56:
Chapter 5 Dynamic Programming In th
Page 57 and 58:
5.2. DISCUSSION: WHY ARE MARKOV POL
Page 59 and 60:
5.3. EXISTENCE OF MINIMIZING SELECT
Page 61 and 62:
5.4. INFINITE HORIZON OPTIMAL CONTR
Page 63 and 64:
5.4. INFINITE HORIZON OPTIMAL CONTR
Page 65 and 66:
5.6. EXERCISES 59 Theorem 5.5.1 The
Page 67 and 68:
5.6. EXERCISES 61 with R, Q, Q T >
Page 69 and 70:
Chapter 6 Partially Observed Markov
Page 71 and 72:
6.3. ESTIMATION AND KALMAN FILTERIN
Page 73 and 74:
6.3. ESTIMATION AND KALMAN FILTERIN
Page 75 and 76:
6.4. PARTIALLY OBSERVED MARKOV DECI
Page 77 and 78:
Chapter 7 The Average Cost Problem
Page 79 and 80:
7.3. LINEAR PROGRAMMING APPROACH TO
Page 81 and 82:
7.3. LINEAR PROGRAMMING APPROACH TO
Page 83 and 84:
7.4. DISCUSSION FOR MORE GENERAL ST
Page 85 and 86:
7.5. EXERCISES 79 7.4.2 Sample-Path
Page 87 and 88:
Chapter 8 Team Decision Theory and
Page 89 and 90:
8.1. EXERCISES 83 u b t = f b (E[x
Page 91 and 92:
Appendix A On the Convergence of Ra
Page 93 and 94:
A.3. CONVERGENCE OF RANDOM VARIABLE
Page 95 and 96:
Bibliography [1] A. Arapostathis, V
show all

Lecture Notes - Department of Mathematics and Statistics - Queen's ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?