Lecture Notes - Department of Mathematics and Statistics - Queen's ...

More documents

Recommendations

Info

16 CHAPTER 3. CLASSIFICATION OF MARKOV CHAINS A set C ⊂ X is said to be communicating if every two elements (states) of C communicate with each other. If every member of the set can communicate to every other member, such a chain is said to be irreducible. The period of a state is defined to be the greatest common divisor of {k > 0 : P k (i, i) > 0}. A Markov chain is called aperiodic if the period of all states is 1. In the following, throughout the notes, we will assume that the Markov process under consideration is aperiodic unless mentioned otherwise. Absorbing Set A set C is called absorbing if P(i, C) = 1 for all i ∈ C. That is, if the state is in C, then the state cannot get out of the set C. A finite set Markov chain in space X is irreducible if the smallest absorbing set is the entire X itself. A Markov chain in X is indecomposable if X does not contain two disjoint absorbing sets. Occupation, Hitting and Stopping Times For any set A ∈ X, the occupation time η A is the number of visits of ψ to set A: η A = ∞∑ t=1 1 {xt∈A}, where 1 (E) denotes the indicator function for an event E, that is, it takes the value 1 when E takes place, and is otherwise 0. Define τ A = min{k > 0 : x k ∈ A}, is the first time that the state visits state i, known as the hitting time. We define also a return time: τ A = min{k > 0 : x k ∈ A, x 0 ∈ A}. A Brief Discussion on Stopping Times The variable τ A defined above is an example for stopping times: Definition 3.1.1 A function τ from a measurable space (X ∞ , B(X ∞ )) to (N + , B(N + )) is a stopping time if for all n ∈ N + , the event {τ = n} ∈ σ(x 0 , x 1 , x 2 , . . . , x n ), that is the event is in the sigma-field generated by the random variables up to time n. Any realistic decision takes place at a time which is measurable. For example if an investor wants to stop investing when he stops profiting, he can stop at the time when the investment loses value, this is a stopping time. But, if an investor claims to stop investing when the investment is at the peak, this is not a stopping time because to find out whether the investment is at its peak, the next state value should be known, and this is not measurable in a causal fashion. One important property of Markov chains is the so-called Strong Markov Property. This says the following: Consider a Markov chain which evolves on a countable set. If we sample this chain according to a stopping time rule, the sampled Markov chain starts from the sampled instant as a Markov chain: Proposition 3.1.1 For a (time-homogenous) Markov chain in a countable state space X, the strong Markov property always holds. That is, the sampled chain itself is a Markov chain; if τ is a stopping time with P(τ
3.1. COUNTABLE STATE SPACE MARKOV CHAINS 17 ∞) = 1, then for any m ∈ N + : for all a, b 0 , b 1 , · · ·. P(x τ+m = a|x τ = b 0 , x τ −1 = b 1 , . . .) = P(x τ+m = a|x τ = b 0 ) = P m (b 0 , a), 3.1.1 Recurrence and Transience Let us define and let us define ∞∑ ∞∑ U(x, A) := E x [ 1 (xt∈A)] = P t (x, A) t=1 t=1 L(x, A) := P x (τ A < ∞), which is the probability of the chain visiting set A, once the process starts at state x. These two are important characterizations for Markov chains. Definition 3.1.2 A set A ⊂ X is recurrent if the Markov chain visits A infinitely often (in expectation), when the process starts in A. This is equivalent to E x [η A ] = ∞, ∀x ∈ A (3.2) That is, if the chain starts at a given state x ∈ A, it comes back to the set A, and does so infinitely often. If a state is not recurrent, it is transient. Definition 3.1.3 A set A ⊂ X is positive recurrent if the Markov chain visits A infinitely often, when the process starts in A and in addition: E x [τ A ] < ∞, ∀x ∈ A Definition 3.1.4 A state α is transient if The above is equivalent to the condition that which in turn is implied by U(α, α) = E α [η α ] < ∞, (3.3) ∞∑ P i (α, α) < ∞, i=1 P i (τ i < ∞) < 1. While discussing recurrence, there is an equivalent, and often easier to check condition: If E i [τ i ] < ∞, then the state {i} is positive recurrent. If L(i, i) = P i (τ i < ∞) = 1, but E i [τ i ] = ∞, then i is recurrent (also called, null recurrent). The reader should connect the above with the strong Markov property: once the process hits a state, it starts from the state as if the past never happened; the process recurs itself. We have the following. The proof is presented later in the chapter, see Theorem 3.3.1. Theorem 3.1.1 If P i (τ i < ∞) = 1, then P i (η i = ∞) = 1. There is a more general notion of recurrence, named Harris recurrence, which is exactly the condition that P i (η i = ∞) = 1. We will investigate this further below while studying uncountable state space Markov chains, however one needs to note that even for countable state space chains Harris recurrence is stronger than recurrence. For a finite state Markov chain it is not difficult to verify that (3.3) is equivalent to L(i, i) < 1, since P(τ i (k) < ∞) = P(τ i (k − 1) < ∞)P(τ i (1) < ∞) and that E[η] = ∑ k P(η ≥ k). If every state in X is recurrent, then the chain is recurrent, if all are transient, then the chain is transient.
Page 1 and 2: i Queen’s University Mathematics
Page 3 and 4: Contents 1 Review of Probability 1
Page 5 and 6: CONTENTS v 5.1 Bellman’s Principl
Page 7 and 8: Chapter 1 Review of Probability 1.1
Page 9 and 10: 1.2. MEASURABLE SPACE 3 1.2.3 Measu
Page 11 and 12: 1.3. PROBABILITY SPACE AND RANDOM V
Page 13 and 14: 1.5. EXERCISES 7 where w t is an in
Page 15 and 16: Chapter 2 Controlled Markov Chains
Page 17 and 18: 2.3. PERFORMANCE OF POLICIES 11 The
Page 19 and 20: 2.4. EXERCISES 13 2.4 Exercises Exe
Page 21: Chapter 3 Classification of Markov
Page 25 and 26: 3.2. STABILITY AND INVARIANT DISTRI
Page 27 and 28: 3.2. STABILITY AND INVARIANT DISTRI
Page 29 and 30: 3.3. UNCOUNTABLE (COMPLETE, SEPARAB
Page 31 and 32: 3.3. UNCOUNTABLE (COMPLETE, SEPARAB
Page 33 and 34: 3.4. FURTHER RESULTS ON THE EXISTEN
Page 35 and 36: 3.5. EXERCISES 29 3.5 Exercises Exe
Page 37 and 38: Chapter 4 Martingales and Foster-Ly
Page 39 and 40: 4.1. MARTINGALES 33 Theorem 4.1.3 I
Page 41 and 42: 4.1. MARTINGALES 35 Theorem 4.1.6 L
Page 43 and 44: 4.2. STABILITY OF MARKOV CHAINS: FO
Page 49 and 50: 4.3. CONVERGENCE RATES TO EQUILIBRI
Page 51 and 52: 4.4. CONCLUSION 45 The second condi
Page 53 and 54: 4.5. EXERCISES 47 ∀ bounded funct
Page 55 and 56: Chapter 5 Dynamic Programming In th
Page 57 and 58: 5.2. DISCUSSION: WHY ARE MARKOV POL
Page 59 and 60: 5.3. EXISTENCE OF MINIMIZING SELECT
Page 61 and 62: 5.4. INFINITE HORIZON OPTIMAL CONTR
Page 63 and 64: 5.4. INFINITE HORIZON OPTIMAL CONTR
Page 65 and 66: 5.6. EXERCISES 59 Theorem 5.5.1 The
Page 67 and 68: 5.6. EXERCISES 61 with R, Q, Q T >
Page 69 and 70: Chapter 6 Partially Observed Markov
Page 71 and 72: 6.3. ESTIMATION AND KALMAN FILTERIN
Page 73 and 74:
6.3. ESTIMATION AND KALMAN FILTERIN
Page 75 and 76:
6.4. PARTIALLY OBSERVED MARKOV DECI
Page 77 and 78:
Chapter 7 The Average Cost Problem
Page 79 and 80:
7.3. LINEAR PROGRAMMING APPROACH TO
Page 81 and 82:
7.3. LINEAR PROGRAMMING APPROACH TO
Page 83 and 84:
7.4. DISCUSSION FOR MORE GENERAL ST
Page 85 and 86:
7.5. EXERCISES 79 7.4.2 Sample-Path
Page 87 and 88:
Chapter 8 Team Decision Theory and
Page 89 and 90:
8.1. EXERCISES 83 u b t = f b (E[x
Page 91 and 92:
Appendix A On the Convergence of Ra
Page 93 and 94:
A.3. CONVERGENCE OF RANDOM VARIABLE
Page 95 and 96:
Bibliography [1] A. Arapostathis, V
show all

Lecture Notes - Department of Mathematics and Statistics - Queen's ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?