Lecture Notes - Department of Mathematics and Statistics - Queen's ...

More documents

Recommendations

Info

6 CHAPTER 1. REVIEW OF PROBABILITY Some examples of commonly encountered random variables are as follows: • Gaussian (N(µ, σ 2 )): f(x) = 1 √ 2πσ e −(x−µ)2 /2σ 2 . • Exponential: • Uniform on [a, b] (U([a, b])): • Binomial (B(n, p)): F(x) = 1 − e −λx , p(x) = λe −λx . F(x) = x − a b − a , p(x) = 1 , x ∈ [a, b]. b − a p(k) = ( n k) p k (1 − p) n−k If n = 1, we also call a binomial variable, a Bernoulli variable. 1.3.3 Independence and Conditional Probability Consider A, B ∈ B(X) such that P(B) > 0. The quantity P(A|B) = P(A ∩ B) P(B) is called the conditional probability of event A given B. This is itself a probability measure. If Then, A, B are said to be independent events. P(A|B) = P(A) A countable collection of events {A n } is independent if for any finitely many sub-collections A i1 , A i2 , . . . , A im , we have that P(A i1 , A i2 , . . . , A im ) = P(A i1 )P(A i2 )...P(A im ). Here, we use the notation P(A, B) = P(A ∩ B). A sequence of events is said to be pairwise independent if for any two pairs (A m , A n ): P(A m , A n ) = P(A m )P(A n ). Pairwise independence is weaker than independence. 1.4 Markov Chains One can define a sequence of random variable as a single random variable living in the product set, that is we can consider {x 1 , x 2 , . . . , x N } as an individual random variable X which is an (E × E × . . . E)−valued random variable in the measurable space (E × E × . . . E, B(E × E × . . . E)), where now the fields are to be defined on the product set. In this case, the probability measure induced by a random sequence is defined on the σ−field B(E × E × . . . E). If the probability measure for this sequence is such that P(x k+1 ∈ A k+1 |x k , x k−1 , . . . , x 0 ) = P(x k+1 ∈ A k+1 |x k ) P.a.s., then {x k } is said to be a Markov chain. Thus, for a Markov chain, the immediate state is sufficient to predict the future; past variables are not needed. As an example, consider the following linear system: x t+1 = ax t + w t ,
1.5. EXERCISES 7 where w t is an independent random variable. This sequence is Markov. Another way to construct a Markov chain is via the following: Let {x t , t ≥ 0} be a random sequence with state space (X, B(X)), and defined on a probability space (Ω, F, P), where B(X) denotes the Borel σ−field on X, Ω is the sample space, F a sigma field of subsets of Ω, and P a probability measure. For x ∈ X and D ∈ B(X), we let P(x, D) := P(x t+1 ∈ D|x t = x) denote the transition probability from x to D, that is the probability of the event {x t+1 ∈ D} given that x t = x. If the probability of the same event given some history of the past (and the present) does not depend on the past, and hence is given by the same quantity, the chain is a Markov chain. Thus, the Markov chain is completely determined by the transition probability and the probability of the initial state, P(x 0 ) = p 0 . Hence, the probability of the event {x t+1 ∈ D} for any t can be computed recursively by starting at t = 0, with P(x 1 ∈ D) = ∑ x P(x 1 ∈ D|x 0 = x)P(x 0 = x), and iterating with a similar formula for t = 1, 2, . . .. The above requires some justification. Kolmogorov’s extension theorem, is one way we can build on to extend the probability measures successively. We will continue our discussion on Markov chains after discussing controlled Markov chains. 1.5 Exercises Exercise 1.5.1 a) Let F β be a σ−field of subsets of some space X for all β ∈ Γ where Γ is a family of indices. Let F = ⋂ β∈Γ F β Show that F is also a σ−field. For a space X, on which a distance metric is defined, the Borel σ−field is generated by the collection of open sets. This means that, the Borel σ−field is the smallest σ−field containing open sets, and as such it is the intersection of all σ-fields containing open sets. On R, the Borel σ−field is the smallest σ−field generated by open intervals. b) Is the set of rational numbers an element of the Borel σ-field on R? Is the set of irrational numbers an element? c) Let X be a countable set. On this set, let us define a metric as follows: { 0,if x = y d(x, y) = 1,if x ≠ y Show that, the Borel σ−field on X is generated by the individual sets {x}, x ∈ X. d) Consider the distance function d(x, y) defined as above defined on R. Is the σ−field generated by open sets according to this metric the same as the Borel σ−field on R? Exercise 1.5.2 a) Let X and Y be real-valued random variables defined on a given probability space. Show that X 2 and X + Y are also random variables. b) Let F be a σ−field of subsets over a set X and let A ∈ F. Prove that {A ∩ B, B ∈ F} is a σ−field over A (that is a σ−field of subsets of A). Exercise 1.5.3 Consider a random variable X which takes values in [0, 1] and is uniformly distributed. We know that for every set S = [a, b) 0 ≤ a ≤ b ≤ 1, the probability of a random variable X taking values in S is P(X ∈ [a, b)) = U([a, b]) = b − a. Consider now the following question? Does every subset S ⊂ [0, 1] admit a probability in the form above? In the following we will provide a counterexample, known as the Vitali set. We can show that a set picked as follows does not admit a measure: Let us define an equivalence class such that x ≡ y if x − y ∈ Q. This equivalence definition divides [0, 1] into disjoint sets. Let A be a subset which picks exactly one element from each equivalent class (here, we adopt what is known as the Axiom of Choice). Since A contains an element from each equivalence class, each pint of [0, 1] is contained in the union ∪ q∈Q (A ⊕ q).
Page 1 and 2: i Queen’s University Mathematics
Page 3 and 4: Contents 1 Review of Probability 1
Page 5 and 6: CONTENTS v 5.1 Bellman’s Principl
Page 7 and 8: Chapter 1 Review of Probability 1.1
Page 9 and 10: 1.2. MEASURABLE SPACE 3 1.2.3 Measu
Page 11: 1.3. PROBABILITY SPACE AND RANDOM V
Page 15 and 16: Chapter 2 Controlled Markov Chains
Page 17 and 18: 2.3. PERFORMANCE OF POLICIES 11 The
Page 19 and 20: 2.4. EXERCISES 13 2.4 Exercises Exe
Page 21 and 22: Chapter 3 Classification of Markov
Page 23 and 24: 3.1. COUNTABLE STATE SPACE MARKOV C
Page 25 and 26: 3.2. STABILITY AND INVARIANT DISTRI
Page 27 and 28: 3.2. STABILITY AND INVARIANT DISTRI
Page 29 and 30: 3.3. UNCOUNTABLE (COMPLETE, SEPARAB
Page 31 and 32: 3.3. UNCOUNTABLE (COMPLETE, SEPARAB
Page 33 and 34: 3.4. FURTHER RESULTS ON THE EXISTEN
Page 35 and 36: 3.5. EXERCISES 29 3.5 Exercises Exe
Page 37 and 38: Chapter 4 Martingales and Foster-Ly
Page 39 and 40: 4.1. MARTINGALES 33 Theorem 4.1.3 I
Page 41 and 42: 4.1. MARTINGALES 35 Theorem 4.1.6 L
Page 43 and 44: 4.2. STABILITY OF MARKOV CHAINS: FO
Page 49 and 50: 4.3. CONVERGENCE RATES TO EQUILIBRI
Page 51 and 52: 4.4. CONCLUSION 45 The second condi
Page 53 and 54: 4.5. EXERCISES 47 ∀ bounded funct
Page 55 and 56: Chapter 5 Dynamic Programming In th
Page 57 and 58: 5.2. DISCUSSION: WHY ARE MARKOV POL
Page 59 and 60: 5.3. EXISTENCE OF MINIMIZING SELECT
Page 61 and 62: 5.4. INFINITE HORIZON OPTIMAL CONTR
Page 63 and 64:
5.4. INFINITE HORIZON OPTIMAL CONTR
Page 65 and 66:
5.6. EXERCISES 59 Theorem 5.5.1 The
Page 67 and 68:
5.6. EXERCISES 61 with R, Q, Q T >
Page 69 and 70:
Chapter 6 Partially Observed Markov
Page 71 and 72:
6.3. ESTIMATION AND KALMAN FILTERIN
Page 73 and 74:
6.3. ESTIMATION AND KALMAN FILTERIN
Page 75 and 76:
6.4. PARTIALLY OBSERVED MARKOV DECI
Page 77 and 78:
Chapter 7 The Average Cost Problem
Page 79 and 80:
7.3. LINEAR PROGRAMMING APPROACH TO
Page 81 and 82:
7.3. LINEAR PROGRAMMING APPROACH TO
Page 83 and 84:
7.4. DISCUSSION FOR MORE GENERAL ST
Page 85 and 86:
7.5. EXERCISES 79 7.4.2 Sample-Path
Page 87 and 88:
Chapter 8 Team Decision Theory and
Page 89 and 90:
8.1. EXERCISES 83 u b t = f b (E[x
Page 91 and 92:
Appendix A On the Convergence of Ra
Page 93 and 94:
A.3. CONVERGENCE OF RANDOM VARIABLE
Page 95 and 96:
Bibliography [1] A. Arapostathis, V
show all

Lecture Notes - Department of Mathematics and Statistics - Queen's ...

Create successful ePaper yourself

Delete template?

Save as template?