Linear Algebra

280 Chapter 3. Maps Between Spaces
Topic: Markov Chains
Here is a simple game. A player bets on coin tosses, a dollar each time, and the
game ends either when the player has no money left or is up to five dollars. If
the player starts with three dollars, what is the chance the game takes at least
five flips? Twenty five flips?
At any point in the game, this player has either $0, or $1, ... , or $5. We
say that the player is the state s0, s1, ... ,ors5. A game consists of moves,
with, for instance, a player in state s3 having on the next flip a .5 chance of
moving to state s2 and a .5 chance of moving to s4. Once in either state s0 or
state s5, the player never leaves that state. Writing pi,n for the probability that
the player is in state si after n flips, this equation sumarizes.
⎛
⎞ ⎛ ⎞ ⎛ ⎞
1 .5 0 0 0 0 p0,n p0,n+1
⎜
⎜0
0 .5 0 0 0⎟⎜p1,n⎟
⎜p1,n+1⎟
⎟ ⎜ ⎟ ⎜ ⎟
⎜
⎜0
.5 0 .5 0 0⎟⎜p2,n⎟
⎜p2,n+1⎟
⎟ ⎜ ⎟
⎜
⎜0
0 .5 0 .5 0⎟⎜p3,n⎟
= ⎜ ⎟
⎜p3,n+1⎟
⎟ ⎜ ⎟ ⎜ ⎟
⎝0
0 0 .5 0 0⎠⎝p4,n⎠
⎝p4,n+1⎠
0 0 0 0 .5 1
p5,n
p5,n+1
For instance, the probability of being in state s0 after flip n +1 is p0,n+1 =
p0,n +0.5 · p1,n. With the initial condition that the player starts with three
dollars, calculation gives this.
⎛
n =0
⎞
0
⎜
⎜0
⎟
⎜
⎜0
⎟
⎜
⎜1
⎟
⎝0
⎠
⎛
n =1
⎞
0
⎜
⎜0
⎟
⎜ .5 ⎟
⎜
⎜0
⎟
⎝ .5⎠
⎛
n =2
⎞
0
⎜ .25 ⎟
⎜
⎜0
⎟
⎜ .5 ⎟
⎝0
⎠
⎛
n =3
⎞
.125
⎜
⎜0
⎟
⎜ .375 ⎟
⎜
⎜0
⎟
⎝ .25 ⎠
⎛
n =4
⎞
.125
⎜ .1875 ⎟
⎜
⎜0
⎟
⎜ .3125 ⎟
⎝0
⎠
···
···
⎛
n =24
⎞
.39600
⎜ .00276 ⎟
⎜
⎜0
⎟
⎜ .00447 ⎟
⎝0
⎠
0 0 .25 .25 .375
.59676
For instance, after the fourth flip there is a probability of 0.50 that the game
is already over — the player either has no money left or has won five dollars.
As this computational exploration suggests, the game is not likely to go on for
long, with the player quickly ending in either state s0 or state s5. (Because a
player who enters either of these two states never leaves, they are said to be
absorbtive. An argument that involves taking the limit as n goes to infinity will
show that when the player starts with $3, there is a probability of 0.60 that the
player eventually ends with $5 and consequently a probability of 0.40 that the
player ends the game with $0. That argument is beyond the scope of this Topic,
however; here we will just look at a few computations for applications.)
This game is an example of a Markov chain, named for work by A.A. Markov
at the start of this century. The vectors of p’s are probability vectors. The
matrix is a transition matrix. A Markov chain is historyless in that, with a
fixed transition matrix, the next state depends only on the current state and
not on any states that came before. Thus a player, say, who starts in state s3,