Lecture Notes - Department of Mathematics and Statistics - Queen's ...

38CHAPTER 4. MARTINGALES AND FOSTER-LYAPUNOV CRITERIA FOR STABILIZATION OF MARKOV CHAINS
Thus, E x0 [τ N − 1 + 1] ≤ V (x 0 ) + b, and by the Monotone Convergence Theorem,
lim
N→∞ E x 0
[τ N ] = E x0 [τ] ≤ V (x 0 ) + b
Now, if we had that
sup V (x) < ∞,
x∈S
the proof would be complete in view of Theorem 3.3.5 and the fact that E x [τ] < ∞ for any x, leading to the
Harris recurrence of S.
Typically, this condition is satisfied. However, in case it is not easy to directly verify, we may need additional
steps to construct a petite set. In the following, we will consider this: Following [23], Chapter 11, define for some
l ∈ Z +
V C (l) = {x : V (x) ≤ l}
We will show that B := V C (l) is itself a petite set, which is recurrent and satisfies the uniform finite-mean-return
property. Since C is petite for some measure ν, we have that
where K a (x, B) = ∑ i a(i)P i (x, B) and hence
K a (x, B) ≥ 1 {x∈C} ν(B), x ∈ X,
1 {x∈C} ≤ 1
ν(B) K a(x, B)
Now,
τ∑
B−1
E x [τ B ] ≤ V (x) + bE x [
1
= V (x) + b
ν(B) E x[
k=0
τ∑
B−1
k=0
1 ∑
τ∑
B−1
= V (x) + b a(i)E x [
ν(B)
i k=0
1 ∑
≤ V (x) + b a(i)(1 + i),
ν(B)
i
τ∑
B−1
1 {xk ∈C}] ≤ V (x) + bE x [
k=0
1
K a (x k , B)] = V (x) + b
ν(B) E x[
1 {xk+i ∈B}]
1
ν(B) K a(x k , B)]
τ∑
B−1
k=0
∑
a(i)P i (x k , B)]
i
where (4.6) follows since at most once the process can hit B between 0 and τ B − 1. Now, the petiteness measure
can be adjusted such that ∑ i a ii < ∞ (by Proposition 5.5.6 of [23]), leading to the result that
sup
x∈B
which can serve as the recurrent set.
This set is furthermore recurrent, since
1 ∑
E x [τ B ] ≤ V (x) + b a(i)(1 + i) < ∞,
ν(B)
sup x∈B E x [τ C ] ≤ sup V (x) + 1 =: K < ∞,
x∈B
and as a result P x (τ C > n) ≤ K/n for any n. Finally, since C is petite, so is B.
There are other versions of Foster-Lyapunov criteria.
i
⋄