STOCHASTIC

W. T. ZIEMBA
the use of the dynamic programming approach. The paper presents a general
framework and a mathematical analysis that attempts to isolate those problems
for which the dynamic programming approach is valid. The exposition
is an adaptation of Denardo and Mitten (1967) that is based in part on some
of their earlier work (see the literature [4, 5, 8]). 2
We will utilize the term sequential decision processes (SDPs) to describe
the phenomenon under study. It will be assumed that the mechanism governing
the evolution of the process is explicit and that the SDPs satisfy certain
termination and monotonicity assumptions. The termination assumption
guarantees that the process is completed in a (fixed) finite number of transitions.
The basic result for such processes is that there exists an optimal policy.
The development also yields an algorithm for determining an optimal policy
and its return function.
The analysis of terminating SDPs requires less formidable mathematical
tools than is required for the analysis of general SDPs. However, the more
complex models, such as infinite horizon models, share many features of the
terminating models. The language used to describe terminating models may
be used for general SDPs. Also the monotonicity property plays a central
role in the analysis of these models as well as of the terminating models.
Furthermore the property of evolution dependence, discussed in Section V,
facilitates the analysis of a large class of infinite horizon models using the
methods discussed here. The development illustrates the central role in
dynamic programming played by the notion of an optimal policy. This notion
may be accepted as a premise called the "optimality postulate" in which it is
assumed that an optimal policy does exist. Utilizing this postulate and the
monotonicity assumption, the functional equations are derived.
II. Sequential Decision Processes
A review of the wide variety of problems that have been treated by dynamic
programming reveals that the vast majority (if not all) may be characterized
as processes that pass through a set of states in response to a sequence of
choices of decisions. The values associated with the process typically depend
on both the states traversed and the decisions made. The following five factors
may be identified as the basic elements of dynamic programming problems:
stages, states, decisions, transitions, and returns. These terms will be given
2 Throughout this book, references cited by numbers in square brackets will be found in
the reference lists at the ends of the articles in which they occur.
44 PART I MATHEMATICAL TOOLS