STOCHASTIC

(c) the p' are independent, and
(d) the p' have discrete distributions.
(e) Attempt to develop a result similar to that in the proposition in the Appendix that
describes a simple optimal policy for case (d).
13. The proofs of Corollaries 1 and 2 of Theorem 1 in the Kalymon paper implicitly
assume that xt remains constant in an optimal policy. Prove rigorously that this is true.
14. (Elementary properties of renewal processes) Let {X„,n = 1,2,...} be a sequence
of nonnegative iid random variables with distribution F, such that P[X„ = G] < 1. Let
ft = EX„ (which exists, but may be oo). Define S0 = 0, Sa = lZ"=iXj, n £ 1, and define
N(t) = sup{n|5„g/}. The stochastic process (N(t), (^0} is called a renewal process.
Intuitively, the X„ are "interarrival times" of some probabilistic process, and N(t) is the
number of "arrivals" in [0,/].
(a) Show that SJn -+ p. w.p. 1 as n -> oo, and N(t) < oo w.p. 1. [Hint: Use the strong
law of large numbers.]
(b) Show that lim,.,*, N{t)/t = 1/// w.p. 1. {Hint: Show that SNm g t g SNm+u and
Af(/)-> oo w.p. 1 as t-* oo.]
Suppose that a reward is earned by the renewal process. Let Y(t) be the total reward earned
by time t, and Y„ the incremental reward earned at the nth renewal. Assume that EX„ and
E\Y„\ are finite, and that the pairs (X„,Y„) are iid. Assume temporarily that Y„ £ 0 and
Y(t) S: 0 is nondecreasing in r.
(c) Show that
£ YJt -> EYJEXt w.p. 1 as / -> oo.
[Hint: YJt = YJN(t) • N(t)jt, and use the strong law of large numbers.]
(d) Show that YNm + y\t -> 0 w.p. 1 as / -> oo.
(e) Show that Y(t)\t -> EYJEXi w.p. 1 as t -* oo.
(f) Prove (e) when the Yn and Y(t) are not restricted in sign.
Properties (e) and (f) will play a fundamental role in the investigation of time averages for
Markov chains in a later exercise. Under the conditions stated above, it is true that (e) and
(f) also hold in the expected value sense: EY(t)/t-* EY1IEXl as t-* oo. The proof of the
latter statement requires the use of more detailed, technical properties of renewal processes
[see Ross (1970)].
A simple but important generalization of the results above pertains to delayed renewal
processes. Let {Xn, n — 1,2,...} be nonnegative independent random variables such that Xi
has distribution G and X2,X3,... have the common distribution F, EX2 = p. Let
So = 0, S„ = 2 Xk, n & 1, and ND(t) = sup{«|5„ g t).
k-l
The process {ND(t), t jg 0} is a delayed renewal process.
(g) Show that ND(t)/t -+ \\p w.p. 1 as / -> oo.
(h) Formulate and prove results similar to (e) and (f) for delayed renewal processes.
15. (Properties of finite Markov chains) Let {X„, « = 0,1,...} be a stationary Markov
chain with finitely many states i=l,2,...,k and stationary one-step transition matrix
690
PART V DYANMIC MODELS