STOCHASTIC

INTRODUCTION TO DYNAMIC PROGRAMMING
Inherent in the proof of the theorem is a procedure for constructing a
policy that is optimal for CI. Indeed, much of the merit of dynamic programming
lies in the efficiency of this algorithm. For the algorithm, it will be convenient
to describe the return function in slightly different notation. Define
the function h(x,dx,va) by
h (x, dx, v6) = vy (x) where yx = dx and yz = 5Z for all z =£ x.
Two notions are present in this definition. First, the dependence on S has
been suppressed; only vs remains. This is justified by the lemma, which assures
that if 5 and y are two policies such that vs = vy, then vn(x) = vx(x) where
n and A are defined by nx = Xx = dx and, for all z # x, nz — bz and kz = yz.
Second, the dependence on dx in h(x,dx,vd) has been made explicit. Note
that h {x, dx, vd) is not a function of 3X—that decision 5X is immaterial, and it
is not required that 5X = dx. In economic terms, h(x,dx,vs) might be interpreted
as the cumulative return obtained by starting at state x and choosing
decision dx with the prospect of receiving the terminating reward vd(z) if
transition occurs to state z. The algorithm is now displayed in the form of a
corollary, whose proof replicates that of the theorem and is left as an exercise
for the reader (Exercise CR.-25).
Corollary 1 Suppose the monotonicity and termination assumptions are
satisfied. Then for each «, the policy