Artificial Intelligence and Soft Computing: Behavioral ... - Arteimi.info

How can we compute the utility value of being in a state? Suppose, if
we reach the goal state, the utility value should be high, say 1. But what
would be the utility value in other states? One simple way to compute static
utility values in a system with the known starting and the goal state is given
here. Suppose the agent reaches the goal S7 from S 1 (fig. 13.13) through a
state say S2. Now we repeat the experiment and find how many times S 2 has
been visited. If we assume that out of 100 experiments, S2 is visited 5 times,
then we assign the utility of state S2 as 5/100=0.05. Further we may assume
that the agent can move from one state to its neighboring state (diagonal
movements not allowed) with an unbiased probability. For example, the agent
can move from S1 to S 2 or S6 (but not to S 5) with a probability of 0.5. If it is in
S5, it could move to S 2, S 4, S 8 or S 6 with a probability of 0.25.
We here make an important assumption on utility.
“The utility of sequence is the sum of the rewards accumulated in the
states of the sequence” [13]. The static utility values are difficult to extract as
it requires large number of experiments. The key to reinforcement learning is
to update the utility values, given the training sequences [13].
In adaptive dynamic programming, we compute utility U(i) of state i by
using the following expression.
U(i) = R(i) + ∑∀j Mij U(j) (13.5)
where R(i) is the reward of being in state i, Mij is the probability of transition
from state i to state j.
In adaptive dynamic programming, we presume the agent to be passive.
So, we do not want to maximize the ∑Mij U(j) term.
For a small stochastic system, we can evaluate the U(i), ∀i by solving
the set of all utility equations like (13.5) for all states. But when the state
space is large, it becomes somewhat intractable.
13.4.3 Temporal Difference Learning
To avoid solving the constraint equations like (13.5), we make an alternative
formulation to compute U(i) by the following expression.
U(i) ← U(i) + α[ R(i ) + (U(j) – U(i) ] (13.6)
where α is the learning rate, normally set in [0, 1].