Lecture Notes - Department of Mathematics and Statistics - Queen's ...

More documents

Recommendations

Info

vi CONTENTS A.3.4 Convergence in Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Chapter 1 Review of Probability 1.1 Introduction Before discussing controlled Markov chains, we first discuss some preliminaries about probability theory. Many events in the physical world are uncertain; that is, with a given knowledge up to a certain time, the entire process is not accurately predictable. Probability theory attempts to develop an understanding for such uncertainty in a consistent way given a number of properties to be satisfied. The suggestion by probability theory may not be physically correct, but it is mathematically precise and consistent. Examples of stochastic processes include: a) The temperature in a city at noon in October 2013: This process takes values in R 31 , b) The sequence of outputs of a communication channel modeled by an additive scalar Gaussian noise when the input is given by x = {x 1 , . . .,x n } (the output process lives in R n ), c) Infinite copies of a coin flip process (living in {H, T } ∞ ), d) The trajectory of a plane flying from point A to point B (taking values in C([t 0 , t v ]), the space of continuous paths in R 3 with x t0 = A, x tf = B), e) The exchange rate between the Canadian Dollar and the American dollar in a given time index T. Some of these processes take values in countable spaces, some do not. When the state space in which a random variable takes values is uncountable, further technical intricacies arise. These are best addressed with a precise characterization of probability and random variables. Hence, probability theory can be used to model uncertainty in the real world in a consistent way according some properties that we expect such measures should admit. The theory may not model the physical world accurately, however. In the following, we will develop a rigorous definition for probability. 1.2 Measurable Space Let X be a collection of points. Let F be a collection of subsets of X with the following properties such that F is a σ-field, that is: • X ∈ F • If A ∈ F, then X \ A ∈ F • If A k ∈ F, k = 1, 2, 3, . . ., then ∞⋃ A k ∈ F k=1 By De Morgan’s laws, the set has to be closed under countable intersections as well. 1
Page 1 and 2: i Queen’s University Mathematics
Page 3 and 4: Contents 1 Review of Probability 1
Page 5: CONTENTS v 5.1 Bellman’s Principl
Page 9 and 10: 1.2. MEASURABLE SPACE 3 1.2.3 Measu
Page 11 and 12: 1.3. PROBABILITY SPACE AND RANDOM V
Page 13 and 14: 1.5. EXERCISES 7 where w t is an in
Page 15 and 16: Chapter 2 Controlled Markov Chains
Page 17 and 18: 2.3. PERFORMANCE OF POLICIES 11 The
Page 19 and 20: 2.4. EXERCISES 13 2.4 Exercises Exe
Page 21 and 22: Chapter 3 Classification of Markov
Page 23 and 24: 3.1. COUNTABLE STATE SPACE MARKOV C
Page 25 and 26: 3.2. STABILITY AND INVARIANT DISTRI
Page 27 and 28: 3.2. STABILITY AND INVARIANT DISTRI
Page 29 and 30: 3.3. UNCOUNTABLE (COMPLETE, SEPARAB
Page 31 and 32: 3.3. UNCOUNTABLE (COMPLETE, SEPARAB
Page 33 and 34: 3.4. FURTHER RESULTS ON THE EXISTEN
Page 35 and 36: 3.5. EXERCISES 29 3.5 Exercises Exe
Page 37 and 38: Chapter 4 Martingales and Foster-Ly
Page 39 and 40: 4.1. MARTINGALES 33 Theorem 4.1.3 I
Page 41 and 42: 4.1. MARTINGALES 35 Theorem 4.1.6 L
Page 43 and 44: 4.2. STABILITY OF MARKOV CHAINS: FO
Page 49 and 50: 4.3. CONVERGENCE RATES TO EQUILIBRI
Page 51 and 52: 4.4. CONCLUSION 45 The second condi
Page 53 and 54: 4.5. EXERCISES 47 ∀ bounded funct
Page 55 and 56: Chapter 5 Dynamic Programming In th
Page 57 and 58:
5.2. DISCUSSION: WHY ARE MARKOV POL
Page 59 and 60:
5.3. EXISTENCE OF MINIMIZING SELECT
Page 61 and 62:
5.4. INFINITE HORIZON OPTIMAL CONTR
Page 63 and 64:
5.4. INFINITE HORIZON OPTIMAL CONTR
Page 65 and 66:
5.6. EXERCISES 59 Theorem 5.5.1 The
Page 67 and 68:
5.6. EXERCISES 61 with R, Q, Q T >
Page 69 and 70:
Chapter 6 Partially Observed Markov
Page 71 and 72:
6.3. ESTIMATION AND KALMAN FILTERIN
Page 73 and 74:
6.3. ESTIMATION AND KALMAN FILTERIN
Page 75 and 76:
6.4. PARTIALLY OBSERVED MARKOV DECI
Page 77 and 78:
Chapter 7 The Average Cost Problem
Page 79 and 80:
7.3. LINEAR PROGRAMMING APPROACH TO
Page 81 and 82:
7.3. LINEAR PROGRAMMING APPROACH TO
Page 83 and 84:
7.4. DISCUSSION FOR MORE GENERAL ST
Page 85 and 86:
7.5. EXERCISES 79 7.4.2 Sample-Path
Page 87 and 88:
Chapter 8 Team Decision Theory and
Page 89 and 90:
8.1. EXERCISES 83 u b t = f b (E[x
Page 91 and 92:
Appendix A On the Convergence of Ra
Page 93 and 94:
A.3. CONVERGENCE OF RANDOM VARIABLE
Page 95 and 96:
Bibliography [1] A. Arapostathis, V
show all

Lecture Notes - Department of Mathematics and Statistics - Queen's ...

Create successful ePaper yourself

Delete template?

Save as template?