STOCHASTIC

W. T. ZIEMBA
Prob(Xt+i£z\(X,,t) = x,dx) is called the probability transition function of
the process and governs its evolution. The transition from stage t to stage
t +1 may be viewed as shown in Fig. 1. The mechanism inside the box may be
Current state (a>, r)
Decision c/x
New state (o/, /+l)
Fig. 1. Transition from stage / to /+1.
thought of as that which chooses a new state from £2(+t via the inputs and the
probability distribution over Clt+l. Implicit in this process is the assumption
that given x = (X,,t) and dx, the random variable Xt+1 is independent of
the random variables Xs,s < t, and their corresponding decisions. Note that
the deterministic case is that special instance when all probability mass in
each stage t falls upon one member of CI,.
A policy 5 is an ordered collection of decisions containing one decision
for each state in Q. The policy space A is the collection of all such policies.
Thus, a policy 5 e A prescribes a particular decision for each and every
state x e £2 and the policy space consists of all possible combinations of
decisions at the various states. The policy space is the Cartesian product of
all the decision sets; that is, A = Ylx e n At • The symbol 5X will frequently
be used to denote the decision in d that applies to state x. Given a particular
policy (5, the process {(X,,t\te T} is Markovian. That is, given the present
state, the futuie evolution of the process is independent of the past. Each
policy d is assumed to have a real-valued return function Vd{x) which reflects
the net reward that would accrue if the process were started in state x and
the specified decisions in