STOCHASTIC

3. DYNAMIC PROGRAMMING
Introduction to Dynamic Programming*
W. T. Ziemba
THE UNIVERSITY OF BRITISH COLUMBIA
I. Introduction
Dynamic programming is a strategy for the solution and analysis of certain
types of optimization problems. The technique has been utilized in many
diverse fields such as production and inventory planning, transportation,
reliability, control theory, economic and physical planning, and design. The
method has also been used in the analysis of many problems in the realm of
recreational mathematics such as puzzles, chess, checkers, blackjack, roulette,
and other games of chance.
The technique of dynamic programming will be of paramount use in the
analysis of the multiperiod financial problems that are considered in Parts IV
and V of this book. This article presents an introduction to the fundamental
ideas, uses, and limitations of dynamic programming. The reading of this
article along with the solution of the associated problems should provide
sufficient background for the study of the material in the sequel.
Generally speaking, dynamic programming is concerned with problems
where an individual must make a sequence of decisions in a specified order.
Problems amenable to analysis by dynamic programming methods must
generally be cast into such a framework even though their original formulation
may not appear to have a sequential nature. Most financial problems, however,
do have a sequential nature where periods of time separate the various
decision points. However, this need not be the case for the application of the
dynamic programming technique.
The underlying mathematical structure of problems amenable to dynamic
programming has been explored by many authors since Richard Bellman
developed the technique in the late 1940s. The basic ideas are even older,
dating at least to Masse in the 1920s. The literature concerned with dynamic
piogramming is positively voluminous—Bellman himself has authored several
hundred papers and more than ten books on the subject. Our intent here is
not to survey this literature 1 but to present a self-contained introduction to
* Adapted from E. V. Denardo and L. G. Mitten, "Elements of Sequential Decision
Processes," Journal of Industrial Engineering 18 (1967), 106-112.
1 For additional information and depth the reader may consult the references listed in
the introduction.
3. DYNAMIC PROGRAMMING 43