Lecture Notes - Department of Mathematics and Statistics - Queen's ...

22 CHAPTER 3. CLASSIFICATION OF MARKOV CHAINS
Now,
||πP − π ′ P || = ||ψP − ψ ′ P ||
= ∑ | ∑ ψ(i)P(i, j) − ∑
j i
k
ψ ′ (k)P(k, j)|
= 1 ∑
||ψ ′ | ∑ ∑
ψ(i)ψ ′ (k)P(i, j) − ψ(i)ψ ′ (k)P(k, j)| (3.5)
|| 1
j k i
≤ 1 ∑∑
∑
||ψ ′ ψ(i)ψ ′ (k)|P(i, j) − P(k, j)| (3.6)
|| 1
j
k
i
= 1 ∑∑
||ψ ′ ψ(i)ψ ′ (k) ∑ |P(i, j) − P(k, j)|
|| 1
k i
j
= 1 ∑∑
||ψ ′ |ψ(i)||ψ ′ (k)|{ ∑ P(i, j) + P(k, j) − 2 min(P(i, j), P(k, j))}
|| 1
k i
j
(3.7)
≤ 1 ∑∑
||ψ ′ |ψ(i)||ψ ′ (k)|(2 − 2δ(P))
|| 1
(3.8)
k
i
= ||ψ ′ || 1 (2 − 2δ(P)) (3.9)
= ||π − π ′ || 1 (1 − δ(P))} (3.10)
In the above, (3.5) follows from adding terms in the summation, (3.6) from taking the norm inside, (3.7) follows
from the relation ||a − b|| = a + b − 2 min(a, b), (3.8) from the definition of δ(P) and finally (3.9) follows from
the l 1 norms of ψ, ψ ′ .
As such, the process P is a contraction mapping if δ(P) > 0. In essence, one proves that such a sequence is
Cauchy, and as every Cauchy sequence in a Banach space has a limit, this process also has a limit. In our setting
{π 0 P m } is a Cauchy sequence. The limit is the invariant distribution. ⊓⊔
Dobrushin’s ergodic theorem provides a guarantee that the limit is unique.
⊓⊔
It should also be noted that Dobrushin’s theorem tells us how fast the sequence of probability distributions
{π 0 P n } converges to the invariant distribution for any arbitrary π 0 .
Ergodic Theorem for Countable State Space Chains
For a Markov chain which has a unique invariant distribution µ(i), we have that almost surely
1
lim
T →∞ T
T∑
t=1
f(x t ) = ∑ i
f(i)µ(i)
∀ f : X → R.
This is called the ergodic theorem due to Birkhoff and is a very powerful result. This is a very important theorem,
because, in essence, this property is what makes the connection with stochastic control, and Markov chains in a
long time horizon. In particular, for a stationary control policy leading to a unique invariant distribution with
bounded costs, it follows that, almost surely,
1
lim
T →∞ T
T∑
c(x t , u t ) = ∑ c(x, u)µ(x, u),
x,u
t=1
∀ real-valued bounded c. The ergodic theorem is what makes a dynamic optimization problem equivalent to a
static optimization problem under mild technical conditions. This will set the core ideas in the convex analytic
approach and the linear programming approach that will be discussed later.