Here - Agents Lab - University of Nottingham

very much tied to the problem at hand and does not necessarily generalise toother related problems. For instance, the learning from four-block problems doesnot generalise to six-block problems and the agent programmer should be awarethat one cannot simply plug-and-play learnt values between problems. While thisfeature may be desirable in many domains, it is nevertheless a shortcoming thatcomes with the ease of use of the programming model that completely insulatesthe programmer from the knowledge representation used for learning.6 Discussion and ConclusionIn this paper we have shown how the mental state representation of an agentprogram may be exploited to significantly increase the effectiveness of the programthrough ongoing learning. The novelty is that this performance improvementcomes almost for free since the programming model remains relativelyunchanged. In particular, we presented an enhancement to the Goal agent programminglanguage that allows adaptive behaviours to be easily programmed.The new language primitive is implemented using a Q-Learning mechanism underthe hood, and allows action choices resulting from programmed rules to beimproved over time based on the ongoing experience of the agent. A key featureof this enhancement is that it can be readily used by agent programmers who arenon-experts in machine learning, since the learning feature has little impact onthe programming model. We demonstrated the usability of the framework in theBlocks World domain and analysed the programmer’s role in balancing betweenfixed and flexible behaviour using three sample solutions for the problem.The results in Section 5, however, also indicate that scalability (i.e. managingthe size of the state space) remains an important challenge. The main tool aprogrammer currently has in our approach to integrating learning into Goal toreduce the state space is to add and exploit knowledge about the environment inthe agent program. Even though the use of domain knowledge may reduce thesize of the state space, which corresponds one-to-one with the number of beliefsand goals of the agent, the state space still quickly becomes very large in theBlocks World environment with an increasing number of blocks [33].We have used and integrated a standard Q-Learning approach to reinforcementlearning. It is well-known that such an approach is unable to handle allbut the smallest state spaces [14]. Our approach, however, does not depend onthis particular choice of learning technique that has been used here mainly todemonstrate the viability of the approach. In order to handle bigger state spacesit is clear that we need some abstraction technique.The ease of use of the new adaptive functionality in Goal is appealing froma programming point of view as shown in this study. The downside is that aprogrammer may waste valuable time in trying to improve performance where itis simply not possible within the constraints of the learning framework and themental state representation used. For example, in a maze world, the only wayto distinguish between two T-junctions that “look” identical is to trace back thehistory of actions that led to the junctions. Here the underlying reinforcementlearning framework is inadequate for learning if the mental state only consists of161