STOCHASTIC

as T-* oo. If {A",} is a stationary process, ergodic arguments may often be applied to obtain
the limit. If {X,} is nonstationary but is asympotically stationary, ergodic arguments will
sometimes be applicable, but may require tedious and technical limiting arguments. In this
problem, an important special case is developed using "elementary" arguments based on
Exercises 14 and 15.
Suppose that {X„, n = 0,1,...} is a stationary, irreducible Markov chain with finitely
many states {\,...,K}, all positive recurrent.
Theorem For any finite, real-valued function/on {l,.,.,K}, and for any starting state
X0 = i,
Km i S/W,) = £;/(*,),
W-.00 VVn=0
where E„ is the expected value with respect to the limiting distribution n.
(a) Prove the theorem when f(k) =1 for i = k and f(k) = 0 otherwise. [Hint: Use
Exercises 14(f) and 15(fc).]
(b) Prove the theorem for general /.
(c) Prove that the theorem holds if the lower limit n = 0 in the sum is replaced by any
fixed finite value.
17. Let {X„, H = 0, 1,...} be a stationary Markov chain with finitely many states and
transition matrix P. Let plj be the probability that X„ = j, given X0 = i, and let f"s be the
probability that starting in j the first transition to j occurs at time n S 1, and let ffi = 0.
Define
PiM = f Pii* a^ FtJ(s) = £ /,>"•
n = 0 rt = 0
(a) Show that Pij(s) and Fu(s) exist for all i,j when \s\ < 1.
(b) Show that
F,j(.s) = P,MIPJJ(S) OV;) and F„(s) = 1/(1-P(((i)).
(c) Show that PtJ(s) satisfies the equations
where
PIJ(S) = 6 u + sJ^p,kPk]{s)
h
lo, i * j.
The expected first passage time Uu from i to y is defined to be the expected time of the first
visit toy, starting from i; that is,
M,J= t,nfft.
n = 0
(d) Show that the utJ satisfy the equations
Ha = 1 + Y. Pikfkjk*l
18. Consider a Bernoulli random walk {A^} which drifts between barriers at 0 and h, such
that the process returns instantly to z (0 < z < h) upon hitting either barrier. Let the probabilities
of positive and negative unit steps be p and q, respectively, where p, q > 0 and
p + q = \.
692
PART V DYNAMIC MODELS