STOCHASTIC

APPENDIX A
with equality holding for an optimal vt. In the limit of continuous time this
condition becomes
0 = max {f{x, v, t) + SeXt v [J(x, ?)]}, (xvii)
V
where S£x> „ is given by an equation like (vii).
Note that the stochastic aspects have dropped out of the problem in this
formulation. Rather than being a random variable X governed by a stochastic
differential equation, the state variable x in Eq. (xvii) is now simply a real
parameter in the control problem. Of course, the dynamics of the original I to
process {X(t)} are reflected in the differential operator S£'. The optimality
condition (xvii) is similar to the Hamilton-Jacobi equation of deterministic
control theory.
There are a number of references which interested readers may find helpful.
For continuous-time stochastic processes, see Breiman (1968), the classic
work of Doob (1953), ltd and McKean (1964), Jazwinski (1970), Kushner
(1967, 1971), Loeve (1963), and Wong (1971). Except for Breiman and Loeve,
these references all contain extensive material on stochastic differential
equations, with Jazwinski (1970) and Kushner (1971) being the most lucid.
For an elementary treatment of deterministic optimal control, see Bellman
(1957, 1961), Kopp (1962), Mangasarian (1966), and Nemhauser (1966). For
a rigorous discussion of the extremely important Pontryagin maximum
principle, see the classic work by Pontryagin et al. (1962), or the equally
rigorous but more lucid new treatment by Boltyanskii (1971). For a discussion
of stochastic optimal control, see Kushner (1967) or the somewhat more lucid
Kushner (1971). In Kushner's treatment, as well as that of the Merton paper
and this Appendix, the class of admissible control functions is limited to
"smooth" cases (differentiable functions, or continuous functions satisfying
a Lipschitz condition). For optimal control problems, it is undesirable to be
limited in this way, since "bang-bang" controls, which are known to be
globally optimal in numerous deterministic problems [e.g., Boltyanskii (1971)],
are artificially eliminated. The theory of stochastic optimal control with a
broader class of admissible controls is quite technical. See Davis and Varaiya
(1973) for a general treatment and references to previous work. The paper of
Vial (1972) previously alluded to describes, heuristically, the application of
discontinuous controls to the optimal cash balance problem.
INTRODUCTION 457