Planning under Uncertainty in Dynamic Domains - Carnegie Mellon ...

2.4. Approaches based on Markov decision processes 15V that is optimal regardless of the starting state [Howard 1960], which satises thefollowing equation:V (s) = max fR(s; a)+X (a; s; u)V (u)gau2STwo popular methods for solving this equation and nding an optimal policy foran mdp are value iteration and policy iteration [Puterman 1994].In policy iteration, the current policy is repeatedly improved by nding someaction in each state that has a higher value than the action chosen by the currentpolicy for that state. The policy is initially chosen at random, and the processterminates when no improvement can be found. Tha algorithm is shown in Table 2.1.This process converges to an optimal policy [Puterman 1994].Policy-Iteration(S; A; ;R;):1. For each s 2 S, (s) = RandomElement(A)2. Compute V (:)3. For each s 2 S f4. Find some action a such thatR(s; a)+ P u2S(a; s; u)V (u) >V (s)5. Set 0 (s) =aif such anaexists,6 otherwise set 0 (s) =(s).g7. If 0 (s) 6= (s) for some s 2 S goto 2.8. Return Table 2.1: The policy iteration algorithmIn value iteration, optimal policies are produced for successively longer nite horizons,until they converge. It is relatively simple to nd an optimal policy over n steps(:), with value function V (:), using the recurrence relation: nn n (s) = argmax afR(s; a)+ X u2S(a; s; u)V n,1 (u)gwith starting condition V 0 (s) =08s2S, where V m is derived from the policy m asdescribed above. Table 2.2 shows the value iteration algorithm, which takes an mdp,a discount value and a parameter and produces successive nite-horizon optimalpolicies, terminating when the maximum change in values between the current andprevious value functions is below . It can also be shown that the algorithm convergesto the optimal policy for the discounted innite case in a number of steps which ispolynomial in jSj, jAj, log max s;a jR(s; a)j and 1=(1 , ).2.4.2 Planning under uncertainty with MDPsThe algorithms described above can nd optimal policies in polynomial time in thesize of the state space of the mdp. However, this state space is usually exponentially