Lecture Notes - Department of Mathematics and Statistics - Queen's ...

18 CHAPTER 3. CLASSIFICATION OF MARKOV CHAINS
3.2 Stability and Invariant Distributions
Stability is an important concept, but it has different meanings in different contexts. In a stochastic setting,
stability means that the state does not grow large too often, and the average over time (temporal) converges to
a statistical average.
If a Markov chain starts at a given time, in the long-run, the chain may forget its initial condition, that is, the
probability distribution at a later time will be less and less dependent on the initial distribution. Given an initial
state distribution, the probability distribution on the state at time 1 is given by:
π 1 = π 0 P
And for t > 1:
π t+1 = π t P = π 0 P t+1
One important property of Markov chains is whether the above iteration leads to a fixed point in the set of
probability measures. Such a fixed point π is called an invariant distribution. A distribution in a countable
state Markov chain is invariant if
π = πP
This is equivalent to
π(j) = ∑ i∈X
π(i)P(i, j), ∀j ∈ X
We note that, if such a π exists, it must be written in terms of π = π 0 lim t→∞ P t , for some π 0 . Clearly, π 0 can
be π itself, but often π 0 can be any initial distribution under irreducibility conditions which will be discussed
further. Invariant distributions are especially important in networking problems and stochastic control, due
to the Ergodic Theorem (which shows that temporal averages converge to statistical averages), which we will
discuss later in the semester.
3.2.1 Invariant Measures via an Occupational Characterization
Theorem 3.2.1 For a Markov chain, if there exists an element i such that E i [τ i ] < ∞; the following is an
invariant measure:
[∑ τi−1
k=0
µ(j) = E
1 ]
x k =j
|x 0 = i , ∀j ∈ X
E i [τ i ]
The invariant measure is unique if the chain is irreducible.
Proof:
We show that
[∑ τi−1
k=0
E
1 ]
x k =j
|x 0 = i
E[τ i ]
= ∑ s
[∑ τi−1
k=0
P(s, j)E
1 ]
x k =s
|x 0 = i
E[τ i ]