Lecture Notes - Department of Mathematics and Statistics - Queen's ...

More documents

Recommendations

Info

14 CHAPTER 2. CONTROLLED MARKOV CHAINS b) Consider the model above. Further, suppose policy D is adopted by the manager. Consider the induced Markov chain by the policy, that is, consider a Markov chain with the following dynamics: with A t as described the the earlier question. Is this Markov chain irreducible? L t+1 = L t + A t − ⌈λ + 0.1⌉N1 (Lt≥⌈λ+0.1⌉)N, Is there an absorbing set in this Markov chain? If so, what is it?
Chapter 3 Classification of Markov Chains 3.1 Countable State Space Markov Chains In this chapter, we first review Markov chains where the state takes values in a finite set or a countable space. A Markov chain {x t } on a countable state space is a collection of random variables {x t , t ∈ Z + }, where each of the variables take place in a countable set X. As such, we will call the collection of such variables a chain Φ which takes values in the (one-sided or two-sided) infinite-product space X ∞ . The Borel σ−field generated on the infinite space is the one generated by the finite dimensional sets of the form {x ∈ X ∞ : x [m,n] = {a m , a m+1 , . . . , a n }, a k ∈ X, m ≤ k ≤ n, m, n ∈ Z + }. We assume that ν 0 is an initial distribution for the Markov chain for x 0 . As such, the process Φ = {x 0 , x 1 , . . .,x n , . . . } is a (time-homogeneous) Markov chain and satisfies: P ν0 (x 0 = a 0 , x 1 = a 1 , x 2 = a 2 , . . .,x n = a n ) = ν(x 0 = a 0 )P(x 1 = a 1 |x 0 = a 0 )P(x 2 = a 2 |x 1 = a 1 )...P(x n = a n |x n−1 = a n−1 ) (3.1) If the initial condition is known to be x 0 , we use P x0 (· · · ) in place of P ν0 (· · · ). We could also write the evolution in terms of a matrix. P(i, j) = Pr(x t+1 = j|x t = i) ≥ 0, ∀i, j ∈ X Here P(·, ·) is also a probability transition kernel, that is for every i ∈ X, P(i, .) is a probability measure on X. Thus, ∑ j P(i, j) = 1. By induction, we could verify that P k (i, j) := P(x t+k = j|x t = i) = ∑ m∈X P(i, m)P k−1 (m, j) In the following, we characterize Markov Chains based on transience, recurrence and communication. We then consider the issue of the existence of an invariant distribution. Later, we will extend the analysis to uncountable space Markov chains. Let us consider a discrete-time, time-homogeneous Markov process living in a countable state space X. Communication If there exists an integer k such that P(x t+k = j|x t = i) = P k (i, j) > 0, and another integer l such that P(x t+l = i|x t = j) = P l (j, i) > 0 then state i communicates with state j. 15
Page 1 and 2: i Queen’s University Mathematics
Page 3 and 4: Contents 1 Review of Probability 1
Page 5 and 6: CONTENTS v 5.1 Bellman’s Principl
Page 7 and 8: Chapter 1 Review of Probability 1.1
Page 9 and 10: 1.2. MEASURABLE SPACE 3 1.2.3 Measu
Page 11 and 12: 1.3. PROBABILITY SPACE AND RANDOM V
Page 13 and 14: 1.5. EXERCISES 7 where w t is an in
Page 15 and 16: Chapter 2 Controlled Markov Chains
Page 17 and 18: 2.3. PERFORMANCE OF POLICIES 11 The
Page 19: 2.4. EXERCISES 13 2.4 Exercises Exe
Page 23 and 24: 3.1. COUNTABLE STATE SPACE MARKOV C
Page 25 and 26: 3.2. STABILITY AND INVARIANT DISTRI
Page 27 and 28: 3.2. STABILITY AND INVARIANT DISTRI
Page 29 and 30: 3.3. UNCOUNTABLE (COMPLETE, SEPARAB
Page 31 and 32: 3.3. UNCOUNTABLE (COMPLETE, SEPARAB
Page 33 and 34: 3.4. FURTHER RESULTS ON THE EXISTEN
Page 35 and 36: 3.5. EXERCISES 29 3.5 Exercises Exe
Page 37 and 38: Chapter 4 Martingales and Foster-Ly
Page 39 and 40: 4.1. MARTINGALES 33 Theorem 4.1.3 I
Page 41 and 42: 4.1. MARTINGALES 35 Theorem 4.1.6 L
Page 43 and 44: 4.2. STABILITY OF MARKOV CHAINS: FO
Page 49 and 50: 4.3. CONVERGENCE RATES TO EQUILIBRI
Page 51 and 52: 4.4. CONCLUSION 45 The second condi
Page 53 and 54: 4.5. EXERCISES 47 ∀ bounded funct
Page 55 and 56: Chapter 5 Dynamic Programming In th
Page 57 and 58: 5.2. DISCUSSION: WHY ARE MARKOV POL
Page 59 and 60: 5.3. EXISTENCE OF MINIMIZING SELECT
Page 61 and 62: 5.4. INFINITE HORIZON OPTIMAL CONTR
Page 63 and 64: 5.4. INFINITE HORIZON OPTIMAL CONTR
Page 65 and 66: 5.6. EXERCISES 59 Theorem 5.5.1 The
Page 67 and 68: 5.6. EXERCISES 61 with R, Q, Q T >
Page 69 and 70: Chapter 6 Partially Observed Markov
Page 71 and 72:
6.3. ESTIMATION AND KALMAN FILTERIN
Page 73 and 74:
6.3. ESTIMATION AND KALMAN FILTERIN
Page 75 and 76:
6.4. PARTIALLY OBSERVED MARKOV DECI
Page 77 and 78:
Chapter 7 The Average Cost Problem
Page 79 and 80:
7.3. LINEAR PROGRAMMING APPROACH TO
Page 81 and 82:
7.3. LINEAR PROGRAMMING APPROACH TO
Page 83 and 84:
7.4. DISCUSSION FOR MORE GENERAL ST
Page 85 and 86:
7.5. EXERCISES 79 7.4.2 Sample-Path
Page 87 and 88:
Chapter 8 Team Decision Theory and
Page 89 and 90:
8.1. EXERCISES 83 u b t = f b (E[x
Page 91 and 92:
Appendix A On the Convergence of Ra
Page 93 and 94:
A.3. CONVERGENCE OF RANDOM VARIABLE
Page 95 and 96:
Bibliography [1] A. Arapostathis, V
show all

Lecture Notes - Department of Mathematics and Statistics - Queen's ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?