Here - Agents Lab - University of Nottingham

for adaptive behaviours remains the same as before; and (iii) learning and programmingbecome truly integrated in the sense that their effectiveness dependsdirectly on the mental state representation used by the programmer.The key idea is that learning may exploit the underspecification that is inherentin agent programming [13]. That is, agent programs often generate multipleoptions for actions without specifying how to make a final choice between theseoptions. This is a feature of agent programming because it does not require aprogrammer to specify action selection to unnatural levels of detail. The motivationof our work is to exploit this underspecification and potentially optimizeaction selection by means of automated learning where it may be too complicatedfor a programmer to optimize code. The first challenge is to add a learningmechanism to agent programming in a generic and flexible way and to naturallyintegrate such a mechanism in a way that burdens the programmer minimally.The second challenge is to do this in such a way that the state space to beexplored by the learning mechanism can still be managed by the program. Ourapproach addresses both challenges by re-using the mental state representationavailable in the agent program. Although our approach also facilitates managingthe state space, there remain issues for future work that need to be dealt within this area in particular. To this end, we draw some lessons learned from ourwork and discuss some options for dealing with this issue.One of the aims of our work is to explore the impact of various representationsor program choices on the learning mechanism. Even though our objectiveis to impose minimal requirements on the programmer’s knowledge of machinelearning, the program structure will have impact on the learning performance.Ideally, therefore, we can give the programmer some guidelines on how to writeagent programs that are able to effectively learn. It is well-known that the representationlanguage is a crucial parameter in machine learning. Given an adequatelanguage, learning will be effective, and given an inadequate one learning willbe difficult if not impossible [14]. Applied to agent-oriented programming thismeans that it is important to specify the right predicates for coding the agent’smental state and to provide the right (modular) program structure to enhancethe effectiveness of learning. If the programmer is not able to use knowledge toguide program design, one may have to search a larger space, may require moreexamples and time, and in the worst case, learning might be unsuccessful.The remainder of the paper is as follows. Section 2 introduces the Goallanguage and reinforcement learning, followed by an overview of related worksin Section 3. Section 4 describes the integration of Goal and reinforcementlearning and Section 5 presents experiments in the Blocks World. We concludewith a discussion of limitations and future directions of this work in Section 6.2 PreliminariesWe now briefly discuss how agent programs and cognitive architectures selectthe action to perform next. In other words, we discuss how the mechanism wewant to extend with a learning capability works. Following this, we introducethe reinforcement learning framework that we have used in this work.149