Linear Algebra, Theory And Applications, 2012a

280 MARKOV CHAINS AND MIGRATION PROCESSES
is in state i. The probability that X n+1 is in state j given that X n is in state i is called
a one step transition probability. When this probability does not depend on n it is called
stationary and this is the case of interest here. Since this probability does not depend on n
it can be denoted by p ij . Here is a simple example called a random walk.
Example 11.3.1 Let there be n points, x i , and consider a process of something moving
randomly from one point to another. Suppose X n is a sequence of random variables which
has values {1, 2, ··· ,n} where X n = i indicates the process has arrived at the i th point. Let
p ij be the probability that X n+1 has the value j given that X n has the value i. SinceX n+1
must have some value, it must be the case that ∑ j a ij =1. Note this says that the sum over
a row equals 1 and so the situation is a little different than the above in which the sum was
over a column.
As an example, let x 1 ,x 2 ,x 3 ,x 4 be four points taken in order on R and suppose x 1
and x 4 are absorbing. This means that p 4k =0forallk ≠ 4 and p 1k =0forallk ≠1.
Otherwise, you can move either to the left or to the right with probability 1 2
. The Markov
matrix associated with this situation is
⎛
⎜
⎝
1 0 0 0
.5 0 .5 0
0 .5 0 .5
0 0 0 1
Definition 11.3.2 Let the stationary transition probabilities, p ij be defined above. The
resulting matrix having p ij as its ij th entry is called the matrix of transition probabilities.
The sequence of random variables for which these p ij are the transition probabilities is called
a Markov chain. The matrix of transition probabilities is called a stochastic matrix.
The next proposition is fundamental and shows the significance of the powers of the
matrix of transition probabilities.
Proposition 11.3.3 Let p n ij denote the probability that X n is in state j given that X 0 was
in state i. Then p n ij is the ijth entry of the matrix P n where P =(p ij ) .
Proof: This is clearly true if n = 1 and follows from the definition of the p ij . Suppose
true for n. Then the probability that X n+1 is at j given that X 0 was at i equals ∑ k pn ik p kj
because X n must have some value, k, and so this represents all possible ways to go from i
to j. You can go from i to1inn steps with probability p i1 and then from 1 to j in one step
with probability p 1j and so the probability of this is p n i1 p 1j but you can also go from i to 2
andthenfrom2toj and from i to 3 and then from 3 to j etc. Thus the sum of these is
just what is given and represents the probability of X n+1 having the value j given X 0 has
the value i.
In the above random walk example, lets take a power of the transition probability matrix
to determine what happens. Rounding off to two decimal places,
⎛
⎜
⎝
1 0 0 0
.5 0 .5 0
0 .5 0 .5
0 0 0 1
⎞
⎟
⎠
20
=
⎛
⎜
⎝
⎞
⎟
⎠ .
1 0 0 0
. 67 9. 5 × 10 −7 0 . 33
. 33 0 9. 5 × 10 −7 . 67
0 0 0 1
Thus p 21 is about 2/3 while p 32 is about 1/3 and terms like p 22 are very small. You see this
seems to be converging to the matrix
⎛ ⎞
1 0 0 0
⎜ 2
1
⎜⎝ 3
0 0 3 ⎟
1
2
3
0 0 ⎠ .
3
0 0 0 1
⎞
⎟
⎠ .