Omar Khalil - Queen's University

State-Dependent Criteria for Convergence of
Markov Chains
Omar khalil
April 18, 2011
1

Contents
1 Introduction 3
2 Background 3
3 Summary of Results 6
4 Critique and Concluding remarks 8
5 References 11
2

1 Introduction
The problem of establishing recurrence and ergodicity of Markov chains is a fundamental
issue in probability theory as well as in the numerous techniques where stochastic stability
of Markov chains plays an important role such as dynamic programming and Markov Chain
Monte Carlo. One of the main techniques of determining whether a Markov chain is stable or
not is the standard Foster-Lyapunov approach which requires that the one-step mean drift of
the chain be negative outside of some appropriately finite set. Although this technique has
proved powerful in analysing many models, as the models get more and more complex analy-
sis of the one-step drift often proves somewhat difficult. Malyˇshev and Men’ˇsikov developed
a refinement of this approach for countable state space chains in [5] allowing the drift to be
negative after a number of steps depending on the initial state.
The goal of this project is to summarize and discuss an important paper [1] by Sean P.
Meyne and R. L. Tweedie which shows that these countable state space results are special
cases of a wider class of chains. The paper also develops a random step approach giving
similar conclusions as well as state dependent drift conditions sufficient to establish geometric
ergodicity of the chain; i.e, the n-step transition probabilities converge to their limits suffi-
ciently geometrically fast.
Although we will attempt to introduce the necessary mathematical concepts as we go by,
a general background in probability theory and the theory of Markov chains is assumed and
one may consult a text such as [9] which covers all the necessary material.
2 Background
Throughout this project we will be working with an underlying measure space (X, F), and a
time homogeneous Markov chain Φ = {Φn, n ∈ Z+}. The transition probabilities of the chain
3

are described by a transition function
with iterates
P : X × F → [0, 1]
(x, A) ↦→ P (x, A)
P n (x, A) = Px(Φn ∈ A), n ∈ Z+, x ∈ X, A ∈ F
We say Φ is ϕ-irreducible if there exists a finite measure ϕ such that for all x ∈ X,
ϕ(A) > 0 ⇒ �
P n (x, A) > 0
n
We will assume for the remainder of our discussions that Φ is ϕ-irreducible. Next we
introduce some minimal definitions of stability concepts.
For A ∈ F define
τA � inf{k ≥ 1 : Φk ∈ A}
The chain Φ is Harris recurrent if for all x ∈ X
ϕ(A) > 0 ⇒ Px(τA < ∞) = 1
A σ-finite measure π on F is called invariant if for all A ∈ F,
�
π(A) =
P (x, A)π(dx)
If Φ is Harris recurrent then a unique invariant measure π exists. If it is finite we say Φ is
positive Harris recurrent. For an aperiodic chain the existence of such a finite π is equivalent
to the condition that
lim
n→∞ ||P n (x, ·) − π|| = 0
for all x ∈ X. We say the chain is geometrically ergodic if this convergence occurs geometri-
cally quickly ; i.e, if there exists a function M and a ρ < 1 such that
||P n (x, ·) − π|| ≤ M(x)ρ n
4

for all x ∈ X and n ∈ Z+.
We mentioned in the introduction that the classical Foster-Lyapunov criteria requires
analysis of the one step mean drift. Some examples of such conditions are the existence of a
suitably unbounded function V and a test set C such that
�
P (x, dy)V (y) ≤ V (x), ∀ x ∈ C c
which along with some other conditions would be equivalent to Harris recurrence. Another
example is Foster’s criterion which requires the existence of some function V ≥ 0 and some
test set C such that
�
P (x, dy)V (y) ≤
� V (x) − 1 if x ∈ C c ,
b < ∞ if x ∈ C
Our last example is a condition for geometric ergodicity for countable space aperiodic chains
which is that there exists a V ≥ 1 and some λ < 1 along with a test set C such that
�
�
λV (x)
P (x, dy)V (y) ≤
b < ∞
if x ∈ Cc ,
if x ∈ C
We say a set C ∈ F is petite if there exists a non-trivial measure ν and a distribution a on
Z+ such that for all x ∈ X and all A ∈ F we have that,
�
a(n)P n (x, A) ≥ ν(A)
n
These sets play the role taken by the test sets in the countable state space context. Last of
our background definitions is what it means for a function V to be unbounded off petite sets
which is that the level sets {V ≤ α} are petite for all α ∈ R.
All of the criteria given above analyse the one step mean drift,
�
∆V (x) = P (x, dy)[V (y) − V (x)]
which proves difficult sometimes, and one may consult [2], [3] or [4] for such situations. Meyn
and Tweedie’s paper contributes by providing criteria which allow analysis of models by using
drifts at times which are not only state dependent but which may in fact be random. We
begin with the result for the fixed state dependent times.
5

3 Summary of Results
Theorem 1. Suppose that Φ is a ϕ-irreducible chain on X and let n(x) be a measurable
function from X to Z+.
(i) The chain is Harris recurrent if there exists a petite set C and a function V ≥ 0 un-
bounded off petite sets such that
�
P n(x) (x, dy)V (y) ≤ V (x), ∀ x ∈ C c
(ii) The chain is positive Harris recurrent if there exists a petite set C and a function V ≥ 0
bounded on C as well as a positive constant b such that
in which case
�
P n(x) (x, dy)V (y) ≤ V (x) − n(x) + bIC(x), ∀ x ∈ X
Ex[τC] ≤ V (x) + b, ∀ x ∈ X
(iii) The chain is geometrically ergodic if it is aperiodic and if there exists some petite set C,
a function V ≥ 1 bounded on C and positive constants λ < 1 and b such that
�
P n(x) (x, dy)V (y) ≤ λ n(x) [V (x) + bIc(x)], ∀x ∈ X (1)
When (1) holds there exist constants r > 1 and R < ∞ such that
�
where π denotes the invariant measure.
n
r n ||P n (x, ·) − π|| ≤ RV (x), ∀x ∈ X
Next we state the main results for random state-dependent times. To analyse those the
paper considers the chain whose next position when Φ0 = x is given by Φζx where ζx is a
random variable with distribution ax on Z+. We define the transition probabilities for this
randomly sampled chain by
�
Ka(x, A) � Px(Φζx ∈ A) = ax(n)P n (x, A)
6
n

We will also define for non-negative functions V the function W given by
W (x) = Ex
�
ζx−1 �
k=0
V (Φk)
We will be referring to two conditions, the first is that
and the second is that
�
W (x) < ∞, ∀ x ∈ X (2)
�
ax(i) ≤ Bxax(k), ∀ k ∈ Z+ (3)
i>k
Examples of distributions satisfying (3) is if ζx is geometric or uniform on {0, 1, . . . , M} for
some M. However this condition does not hold if ζx is degenerate at n(x) which is the
situation in Theorem 1. We will now state the random time results.
Theorem 2. Suppose that Φ is a ϕ-irreducible chain on X and let ζx be a random variable
on Z+, for each x independent of the chain.
(i) The chain is Harris recurrent if there exists a petite set C and a function V ≥ 0 un-
bounded off petite sets such that
�
Ka(x, dy)(x, dy)V (y) ≤ V (x), ∀ x ∈ C c
(ii) The chain is positive Harris recurrent if there exists a petite set C and a function V ≥ 0
bounded on C as well as a positive constant b such that
provided (2) holds; in which case,
�
Ka(x, dy)V (y) ≤ V (x) − 1 + bIC(x), ∀ x ∈ X (4)
Ex[τC] ≤ W (x) + b, ∀ x ∈ X
If ax satisfies (3) and (4) holds then (2) holds.
7

(iii) The chain is geometrically ergodic if it is aperiodic and if there exists some petite set C,
a function V ≥ 1 bounded on C and positive constants λ < 1 and b such that
�
Ka(x, dy)(x, dy)V (y) ≤ λV (x) + bIc(x), ∀x ∈ X
provided ax satisfies (3) with Bx ≤ B < ∞ independent of x. In which case there exist
constants r > 1 and R < ∞ such that
�
n
r n ||P n (x, ·) − π||V ≤ RV (x), ∀x ∈ X
where π denotes the invariant measure and the V -norm || · ||V is defined for any signed
measure µ by
�
||µ||V � sup
|g|≤V
4 Critique and Concluding remarks
µ(dy)g(y)
This paper by Meyne and Tweedie introduces several contributions to the existing literature.
The first is that it generalizes the results of Malyˇshev and Men’ˇsikov in [5] to the wider
context of ϕ-irreducible chains as well as providing sample path proofs which fit nicely with
the nature of such chains. The second is the development of the random step approach which
we have seen earlier in the summary of the results. Finally the paper contributes by finding
state dependent sufficient conditions for the recurrence and ergodicty of Markov chains that
provides an alternative to analysing the one step mean drift. This is particularly useful when
drift functions which depend on the whole history of the chain are used such as in [2],since
in those cases analysis of the one step mean drift becomes complicated.
A natural question however which arises upon reading this paper is what about the random
times which don’t meet the geometric growth condition (3) ? It seems plausible to assume
that both theorems 1 and 2 are special cases of a more general result covering random times
which don’t necessarily meet that condition and in fact later extensions such as [6] and [7]
, show that it is possible to let go of that condition and analyse random times whose tail is
8

not necessarily geometric in nature. Meyne displays however that a hopeful jump such as
replacing the condition in theorem 1 (ii) with
instead of
�
�
P n(x) (x, dy)V (y) ≤ V (x) − 1 + bIC(x)
P n(x) (x, dy)V (y) ≤ V (x) − n(x) + bIC(x)
is not possible by providing a counter example. Hence the correct generalization is not nec-
essarily obvious.
An element of the paper which we did not have a chance to cover in the summary due
to space constraints but would still like to give a general idea about are the applications,
both mentioned in the paper itself and the ones which depend on state dependent criteria in
general. It is useful to note that before this paper the applications which relied only on the
countable state results developed in [5] have typically been on orthants where the negative
drift could be established easily in the interior of the space and drift with a fixed number of
steps established on the boundary of the space. However even in those cases Meyn shows that
the state dependent criteria provided in this paper could make things simpler. Nonetheless
the paper exhibits both discrete time and continuous time applications where analysis using
the fixed state criteria would be difficult. The applications included an antibody model, a
non-linear autoregressive model as well as complex queuing models.
Concerning applications in general, the state dependent criteria technique is the basis of
the fluid model approach to stability in stochastic networks and other general models. Fur-
thermore, discussions in [7] show that the state dependent random sampling technique has
applications in networked control, where there is only intermittent access to either sensor
information or control and depending on the availability of resources the timing is in general
random. Hence the results of the paper prove applicable in numerous settings beyond pure
probability theory.
In conclusion this paper not only contributes important sufficiency results but also introduces
9

useful techniques for analysing stochastic stability of Markov chains in general. The state de-
pendent random sampling criteria proves to have many practical applications. Natural steps
beyond this paper include generalizing theorems 1 and 2 to results where both these theorems
are special cases and which do not necessarily include the geometric growth condition. Fur-
thermore one may want to consider analysing the rates of convergence of the Markov chains
which is of particular importance in real time control systems.
10

5 References
[1] Meyn, S.P. and Tweedie, R. (1994) State-dependent criteria for convergence of Markov
chains Ann. Appl. Probab. 4 149-168.
[2] Meyn, S. P. and Guo, L. (1992). Stability, convergence, and performance of an adaptive
control algorithm applied to a randomly varying system. IEEE Trans. Automat. Control
AC-37 535-540.
[3] Chen, R and Tsay, R.S. (1991). On the ergodicity of TAR(1) processes. Ann. Appl.
Probab. 1 613-634.
[4] Tjøsthem, D. (1990). Non–linear time series and Markov chains. Adv. in Appl. Probab.
22 587-611.
[5] Malyˇshev, V. A. and Men’ˇsikov, M. V. (1982). Ergodicity, continuity and analyticity of
countable Markov chains. Trans. Moscow Math. Soc. 1 1-48.
[6] Yüksel, S. (2009). A Random Time Stochastic Drift Result and Application to Stochastic
Stabilization Over Noisy Channels. Allerton Conference, Illinois, USA.
[7] Yüksel, S. and Meyn. S. P. (2010). Random-Time, State-Dependent Stochastic Drift for
Markov Chains and Application to Stochastic Stabilization Over Erasure Channels. IEEE
Trans. Automat. Control.
[8] Yüksel, Serdar. (2011). Math 472/872 Course Lecture Notes. Queen’s University.
[9] It ô, Kiyosi. Introduction to Probability Theory. Cambridge University Press, 1986.
11