Homework #4 Solutions (pdf file)

9. (Ross 4.63)⎛ ⎞.4 .2 .1P T = ⎝ .1 .5 .2⎠.3 .4 .2⎛⎞−1⎛⎞0.6 −0.2 −0.1 2.2069 1.3793 0.6207S = (I−P T ) −1 = ⎝−0.1 0.5 −0.2⎠= ⎝0.9655 3.1034 0.8966⎠−0.3 −0.4 0.8 1.3103 2.0690 1.9310Sof 13 = S 13S 33= 0.3214f 23 = S 23S 33= 0.4643f 33 = S 33 − 1S 33= 0.482110. ( Electronic reserves #15)(a) For states 1 pint, 2 pints, ..., 7 pints, the transition matrix is⎛⎞0.6 0.40.3 0.3 0.40.1 0.2 0.3 0.4P =0.1 0.2 0.3 0.4⎜ 0.1 0.2 0.3 0.4⎟⎝0.1 0.2 0.3 0.4⎠0.1 0.2 0.7(b) Fromwe getπ 1 = 0.6π 1 + 0.3π 2 + 0.1π 3 ;π 2 = 0.4π 1 + 0.3π 2 + 0.2π 3 + 0.1π 4 ;.π 7 = 0.4π 6 + 0.7π 7 ;π 1 + π 2 + · · · + π 7 = 1.π 1 = 0.139; π 2 = 0.139; π 3 = 0.139; π 4 = 0.138; π 5 = 0.141; π 6 = 0.130; π 7 = 0.174.4

11. (Electronic reserves #16)(a)Not to advertise:P 1 =( 3/4) 1/41/2 1/2Always advertise:Manager’s proposal:P 2 =P 3 =( 1/2) 1/21/4 3/4( 1/2) 1/21/2 1/2(b)Not to advertise:getπ 0 = 3/4π 0 + 1/2π 1π 1 = 1/4π 0 + 1/2π 1π 0 + π 1 = 1π 0 = 2/3, π 1 = 1/3;Similarly for ”Always advertise”:π 0 = 1/3, π 1 = 2/3;Manager’s proposal:π 0 = 1/2, π 1 = 1/2;(c) profit when never advertise: 2 ∗ 2/3 + 4 ∗ 1/3 = 8/3 million;profit when always advertise: 2 ∗ 1/3 + 4 ∗ 2/3 − 1 = 7/3 million;profit of manager’s proposal: 2 ∗ 1/2 + 4 ∗ 1/2 − 1 ∗ 1/2 = 5/2 million.Hence, the strategy of never advertising is the most profitable.12. (Electronic reserves #24)(a) State 0 means the recorder has been replaced, State 1 means themachine is new, State 2 means the machine works well through the wholefirst year, State 3 means the machine works well for the whole first 2 years,and does not need repair. The transition matrix is⎛⎞1 0 0 0P = ⎜.01 0 .99 0⎟⎝.05 0 0 .95⎠0 0 0 15

(b) This is the probability that the recorder will fail either during the firstyear or the second year: .01 + .99 ∗ .05 = .0595.13. (Extra Credit: Ross 3.26)Consider the process defined by the successive card orderings, so there aren! states. The one-step transition probabilities of this process areP (i1,...,i n),(i j,i 1,...,i j−1,i j+1,...,i n) = 1 nObviously, there are n equally possible resulting states in each transition,i.e., each row of the corresponding Transition Probability Matrix (TPM)has n entries equal to 1/n. However, the suggested shuffling further impliesthat each ordering (i 1 , ..., i n ) can be reached in one transition from exactlyn different orderings, specifically, those orderings obtained from (i 1 , ..., i n )by reolcating card i 1 to the i th position, for i = 1, 2, . . . , n. Hence, eachcolumn of the considered TPM will have exactly n non-zero entries, andsince each such entry is equal to 1/n, the matrix is doubly stochastic.But then, the result off Exercise 4.20 implies that, in the limit, all n! possiblestates are equally likely.14. (Extra Credit: Ross 3.37)We need to show thatπ j = ∑ iπ i P k i,jBut this equation is an immediate consequence of the facts that (i) P k i,j arethe k-step transition probabilities for the original Markov chain definedby [P i,j ], and (ii) {π j } is a stationary distribution for that chain.15. (Extra Credit: Ross 3.49)(a)This is not a Markov chain, because the process transition at the n thepoch is not determined only by its current state but also by the result ofthe coin’s flipping at the very beginning.For the considered process,lim P (X n = i) = pπ 1 (i) + (1 − p)π 2 (i).n→∞(b)This is a Markov chain, since for this case, we will flip the coin atthe beginning of each epoch, and these trials are independent. Since theoriginal stochastic processes are Markov chains, the composite process willalso satisfy the Markovian property.The one-step transition probability matrix for this process is given byP ij = pP (1)ij + (1 − p)P (2)ij .6

The following counter-example establishes that the limiting distributionsfor the two above cases may not be the same:LetP (1) =( 1/2) 1/21/2 1/2Then, its limiting distribution is π 1 (1) = 1/2, π 1 (2) = 1/2.Also, letP (1) =(1/4) 3/41/2 1/2The corresponding limiting distribution is π 2 (1) = 2/3, π 2 (2) = 1/3.Finally, let also p = 1/2.Then, according to the above,lim P (X n = 1) = 1/2 · 1/2 + 1/2 · 2/3 = 7/12n→∞On the other hand, the transition matrix for the second process is( ) 3/8 5/8P =1/2 1/2with limiting probability for State 1 equal to 4/9.7