guidance, flight mechanics and trajectory optimization

(2.4.12) Some Limitations on the Development of the Bellman Equation
The preceding paragraphs of this section have been primarily concerned
with reducing the computational algorithm inherent in the Principle of
C@timality to a certain partial differential equation called the Bellman
equation. From this equation various additional properties of the optimal
decision sequence have been developed and shown to be equivalent to the
necessary conditions normally developed by means of the Calculus of Variations
or the Maximum Principle. In some special cases, however, the Bellman
equation, which results from considering the Principle of Optimality in the
limit as the separation between states and decision goes to zero, is
erroneous.
In developing the Bellman equation which, for the Bolza probleln, took
the form
it was necessary to assume that all second derivatives of R exist and are
bounded (see Eq. 2.4.115) which implies, among other things, that all
first derivatives of R exist and are continuous. It is shown in Ref. (2.4.3)
that occasionally the derivatives 2% do not exist at all points in the
(t, x) space and hence, that Eq. (2.4'.166) is not always correct. The
type of problem in which this may happen is one in which the control action
appears linearly in the state equations; that is, the state equations take
the form
(2.4.166)
(2.4.167)
with the result that the optimal control is bang-bang in that it jumps
discontinuously from one boundary of the control set 7-f to another boundary.
If there exists a curve in the (x, t) space (called a switching curve) with
the property that all optimal trajectories when striking the curve experience
a control discontinuity, and if furthermore a finite segment o$Ethe optimal
solution lies along the switching curve, then the derivatives - may
not exist along the switching curve and Eq. (2.4.166) may not f%'applicable.
As an example of such a problem, consider the second order integrator
% = x2
?2 =U
105
(2.4.168)