From Algorithms to Z-Scores - matloff - University of California, Davis

20.1. DISCRETE-TIME MARKOV CHAINS 401
Intuitively, the existence of πi implies that as t approaches infinity, the system approaches steadystate,
in the sense that
lim
t→∞ P (Xt = i) = πi
(20.4)
Actually, the limit (20.4) may not exist in some cases. We’ll return to that point later, but for
typical cases it does exist, and we will usually assume this.
20.1.2.1 Derivation of the Balance Equations
Equation (20.4) suggests a way to calculate the values πi, as follows.
First note that
P (Xt+1 = i) =
P (Xt = k and Xt+1 = i) =
P (Xt = k)P (Xt+1 = i|Xt = k) =
P (Xt = k)pki
k
k
k
(20.5)
where the sum goes over all states k. For example, in our random walk example above, we would
have
P (Xt+1 = 3) =
5
P (Xt = k and Xt+1 = 3) =
k=1
5
P (Xt = k)P (Xt+1 = 3|Xt = k) =
k=1
Then as t → ∞ in Equation (20.5), intuitively we would have
πi =
k
πkpki
5
P (Xt = k)pk3
k=1
(20.6)
(20.7)
Remember, here we know the pki and want to find the πi. Solving these balance equations
equations (one for each i), gives us the πi.
For the random walk problem above, for instance, the solution is π = ( 1 3 3 3 1
11 , 11 , 11 , 11 , 11 ). Thus in
the long run we will spend 1/11 of our time at position 1, 3/11 of our time at position 2, and so
on.