Planning under Uncertainty in Dynamic Domains - Carnegie Mellon ...

3.3. A Markov decision process model of uncertainty 31(at barge1 Richmond)(operational barge1)(poor-weather)True in all states:(distance Richmond west-coast 1)(ready barge1)0.5 0.250.167 0.083(at barge1 west-coast) (at barge1 west-coast) (at barge1 west-coast) (at barge1 west-coast)(operational barge1)(operational barge1)(poor-weather)(poor-weather)(fair-weather)(fair-weather)Figure 3.7: The set of states reachable when the action Move-Barge is executedfrom the initial state shown, when Weather-Brightens is the only exogenous eventin the domain and the action has a duration of one time unit. The probability ofreaching each state is shown along its arc.3.3 A Markov decision process model of uncertainty.In the previous section I showed the result when an action is applied and one eventwith the same duration may take place simultaneously. In general, actions and eventsmay have dierent durations and events may take eect after an action begins butbefore it ends. Multiple events may take place at the same time, and they may haveeects that alter the same literals in dierent ways. In order to precisely state theprobability distributions of state histories that can result from a sequence of actionsin a planning problem, I introduce its underlying Markov decision process (MDP).Recall from Chapter 2, on related work, that an MDP is dened by its statespace, its state transition function and its reward function. A state in the state spaceof the underlying MDP M contains more information than a state in the planningproblem, since it contains a limited history of actions and exogenous events that arein progress. This is because the MDP models the interaction of actions and eventsthat have dierent lengths and whose time intervals may overlap, and it is useful tochoose the state space and transition function such that each successor state occurs axed unit of time after the predecessor state. For this reason, a state in M contains ahistory of the events and actions taking place in the world as well as the current statedescribed in terms of literals from the planning problem. Individual state transitionsthen correspond to the passage of a unit of time rather than an action or event viewedas an atomic unit.The action and event history contained in a state in M consists of a set of pairsn: where n is an integer denoting the time before the action or event will take place,and is an eect set, a list of add and delete statements that specify the changes to