Artificial Intelligence and Soft Computing: Behavioral ... - Arteimi.info

In the last expression, we updated U(i) by considering the fact that we
should allow transition to state j from state i, when U(j) >> U(i). Since we
consider temporal difference of utilities, we call this kind of learning temporal
difference (TD) learning.
It seems that when a rare transition occurs from state j to state i, U(j)-
U(i) will be too large, causing U(i) large by (13.6). However, it should be kept
in mind that the average value of U(i) will not change much, though its
instantaneous value seems to be large occasionally.
13.4.4 Active Learning
For passive learner, we considered M to be a constant matrix. But for an
active learner, it must be a variable matrix. So, we redefine the utility
equation of (13.5) as follows.
U(i) = R(i) + Maxa∑∀j Mij a U(j) (13.7)
where Mij a denotes the probability of reaching state j through an action ‘a’
performed at state i. The agent will now choose the action a for which Mij a is
maximum. Consequently, U(i) will be maximum.
13.4.5 Q-Learning
In Q-learning, instead of utility values, we use q-values. We employ Q(a, i) to
denote the Q-value of doing an action a at state i. The utility values and Qvalues
are related by the following expression:
U(i) = maxa Q(a, i). (13.8)
Like utilities, we can construct a constraint equation that holds at equilibrium,
when the Q-values are correct [13].
Q(a, i) = R(i) + ∑Mij a .maxa’ Q(a’,i). (13.9)
The corresponding temporal-difference updating equation is given by
Q(a, i) ← Q(a, i) + α[ R(i) + max(a’, j) – Q(a, i) ] (13.9(a))
which is to be evaluated after every transition from state i to state j.
The Q-learning continues following expression 13.9 (a) until the Q-values at
each state i in the space reaches a steady value.