STOCHASTIC

INTRODUCTION TO DYNAMIC PROGRAMMING
such that T"*(x) = 0. Naturally nx may vary with x. The termination condition
implies that for each x e Q, nx is the smallest positive integer such that
r»(x) = 0foralln^«x.
For m 2: 1, let Sm be the subset of Q containing those points from which
termination always occurs in m transitions or less but does not always occur
in fewer than m transitions. That is, let Sm = {x e Q. | nx=m}. The termination
assumption implies that Q= \J%=1Sm. Similarly, let S m be the set of all
states in Q. from which termination always occurs in m transitions or less;
that is, let 5"" = (Ji"=1 St. Should there exist an integer N such that nx ^ N
for every xeQ and nx = N for at least one x, then the process is called an
N-stage sequential decision process. N-stage processes cover a wide variety
of important applications.
The term "regeneration point" arises since the process is Markovian.
Hence the probability law governing transitions from a given state is independent
of what states preceded it. Here, the return vd(x) does not depend on
how state x was attained. Note that {x} u Qx contains x and all states which
are accessible from x. The following lemma demonstrates that v5 (x) depends
only on those decisions in