Lecture Notes - Department of Mathematics and Statistics - Queen's ...

42CHAPTER 4. MARTINGALES AND FOSTER-LYAPUNOV CRITERIA FOR STABILIZATION OF MARKOV CHAINS
Observe the difference with the inf-compactness condition leading to recurrence and the above condition, leading
to non-recurrence.
We finally note that, a convenient way to verify instability or transience is to construct an appropriate martingale
sequence.
4.2.6 State Dependent Drift Criteria: Deterministic and Random-Time
It is also possible that, in many applications, the controllers act on a system intermittently. In this case, we
have the following results [33]. These extend the deterministic state-dependent results presented in [23], [24]:
Let τ z , z ≥ 0 be a sequence of stopping times, measurable on a filtration, possible generated by the state process.
Theorem 4.2.7 [33] Suppose that x is a ϕ-irreducible Markov chain. Suppose moreover that there are functions
V : X → (0, ∞), δ: X → [1, ∞), f : X → [1, ∞), a petite set C on which V is bounded, and a constant b ∈ R,
such that the following hold:
E[V (x τz+1 ) | F τz ] ≤ V (x τz ) − δ(x τz ) + b1 {xτz ∈C}
[ τz+1−1 ∑ ]
E f(x k ) | F τz ≤ δ(x τz ), z ≥ 0.
k=τ z
Then X is positive Harris recurrent, and moreover π(f) < ∞, with π being the invariant distribution.
(4.9)
⊓⊔
By taking f(x) = 1 for all x ∈ X, we obtain the following corollary to Theorem 4.2.7.
Corollary 4.2.2 [33] Suppose that X is a ϕ-irreducible Markov chain. Suppose moreover that there is a function
V : X → (0, ∞), a petite set C on which V is bounded,, and a constant b ∈ R, such that the following hold:
Then X is positive Harris recurrent.
E[V (x τz+1 ) | F τz ] ≤ V (x τz ) − 1 + b1 {xτz ∈C}
sup E[τ z+1 − τ z | F τz ] < ∞.
z≥0
(4.10)
⊓⊔
More on invariant probability measures
Without the irreducibility condition, if the chain is weak Feller, if (4.5) holds with S compact, then there exists
at least one invariant probability measure as discussed in Section 3.4.
Theorem 4.2.8 [33] Suppose that X is a Feller Markov chain, not necessarily ϕ-irreducible. Then,
If (4.9) holds with C compact then there exists at least one invariant probability measure. Moreover, there
exists c < ∞ such that, under any invariant probability measure π,
∫
E π [f(x)] = π(dx)f(x) ≤ c. (4.11)
4.3 Convergence Rates to Equilibrium
X
In addition to obtaining bounds on the rate of convergence through Dobrushin’s coefficient, one powerful approach
is through the Foster-Lyapunov drift conditions.
Regularity and ergodicity are concepts closely related through the work of Meyn and Tweedie [25], [26] and
Tuominen and Tweedie [30].