Lecture Notes - Department of Mathematics and Statistics - Queen's ...

40CHAPTER 4. MARTINGALES AND FOSTER-LYAPUNOV CRITERIA FOR STABILIZATION OF MARKOV CHAINS
Corollary 4.2.1 In particular, under the conditions of Theorem 4.2.3,
∫
∫
π(dx)f(x) =
lets one obtain a bound on the expectation of f(x).
S
τ∑
A−1
π(dx)E x [ f(x t )] ≤
t=0
∫
A
π(dx)E x [V (x) + b].
See chapter 14 of [23] for further details.
We need to ensure that there exists an invariant measure however. This is why we require that f : X → [1, ∞),
where 1 can be replaced with any positive number.
4.2.3 Criterion for Recurrence
Theorem 4.2.4 (Foster-Lyapunov for Recurrence) Let S be a compact set, b < ∞ and V (.) be an infcompact
functional on X such that for all α ∈ R + {x : V (x) ≤ α} is compact. Let {x n } be an irreducible Markov
chain on X. If the following is satisfied:
∫
P(x, dy)V (y) ≤ V (x) + b1 x∈S , ∀x ∈ X , (4.6)
X
then, with τ S = min(t > 0 : x t ∈ S), P x (τ S < ∞) = 1 for all x ∈ X, that is the Markov chain is Harris recurrent.
Proof: Let τ S = min(t > 0 : x t ∈ S). Define two stopping times: τ S and τ BN where B N = {x : V (x) ≥ N}. Note
that a sequence defined by M t = V (x t ) (which behaves as a supermartingale for t = 0, 1, · · · , min(τ S , τ BN ) − 1)
is uniformly integrable until τ BN , and a variation of the optional sampling theorem applies for stopping times
which are not necessarily bounded by a given finite number with probability one. Note that, due to irreducibility,
min(τ S , τ BN ) < ∞ with probability 1. Now, it follows that for x /∈ S ∪ B N , since when exiting into B N , the
minimum value of the Lyapunov function is N:
for some finite positive M. Hence,
V (x) = E x [V (x min(τS,τ BN ))] ≥ P x (τ BN < τ S )N + P x (τ BN ≥ τ S )M,
P x (τ BN < τ S ) ≤ V (x)/N
We also have that P(min(τ S , τ BN ) = ∞) = 0, since the chain is irreducible and it will escape any compact set in
finite time. As a consequence, we have that
P x (τ S = ∞) ≤ P(τ BN < τ S ) ≤ V (x)/N
and taking the limit as N → ∞, P x (τ S = ∞) = 0.
⊓⊔
If S is further petite, then once the petite set is visited, any other set with a positive measure (under the
irreducibility measure) is visited with probability 1 infinitely often.
⋄
4.2.4 On small and petite sets
Establishing petiteness may be difficult to directly verify. In the following, we present two conditions that may
be used to establish the petiteness properties.
By [23], p. 131: For a Markov chain with transition kernel P and K a probability measure on natural numbers,
if there exists for every E ∈ B(X), a lower semi-continuous function N(·, E) such that ∑ ∞
n=0 P n (x, E)K(n) ≥
N(x, E), for a sub-stochastic kernel N(·, ·), the chain is called a T −chain.
Theorem 4.2.5 [23] For a T −chain which is irreducible, every compact set is petite.