From Algorithms to Z-Scores - matloff - University of California, Davis

422 CHAPTER 20. MARKOV CHAINS
20.5.4 Example: Finite Random Walk
Let’s go back to the example in Section 20.1.1.
Suppose we start our random walk at 2. How long will it take to reach state 4? Set bi =
E(Ti4|start at i). From (20.62) we could set up equations like
b2 = 1
3 (1 + b1) + 1
3 (1 + b2) + 1
3 (1 + b3) (20.71)
Now change the model a little, and make states 1 and 6 absorbing. Suppose we start at position
3. What is the probability that we eventually are absorbed at 6 rather than 1? We could set up
equations like (20.65) to find this.
20.5.5 Example: Tree-Searching
Consider the following Markov chain with infinite state space {0,1,2,3,...}. 8 The transition matrix
is defined by pi,i+1 = qi and pi0 = 1 − qi. This kind of model has many different applications,
including in computer science tree-searching algorithms. (The state represents the level in the tree
where the search is currently, and a return to 0 represents a backtrack. More general backtracking
can be modeled similarly.)
The question at hand is, What conditions on the qi will give us a positive recurrent chain?
Assuming 0 < qi < 1 for all i, the chain is clearly irreducible. Thus, to check for recurrence, we
need check only one state, say state 0.
For state 0 (and thus the entire chain) to be recurrent, we need to show that P (T00 < ∞) = 1. But
Therefore, the chain is recurrent if and only if
P (T00 > n) = Π n−1
i=0 qi
(20.72)
lim
n→∞ Πn−1
i=0 qi = 0 (20.73)
For positive recurrence, we need E(T00) < ∞. Now, one can show that for any nonnegative integer-
8 Adapted from Performance Modelling of Communication Networks and Computer Architectures, by P. Harrison
and N. Patel, pub. by Addison-Wesley, 1993.