Here - Agents Lab - University of Nottingham

2.1 Agent Programming LanguagesAgent programming languages (APLs) based on the BDI paradigm are rulebasedlanguages [15, 16]. Rules may serve various purposes but are used amongothers to select the actions or plans that an agent will perform. An agent programmay perform built-in actions that are provided as programming constructs thatare part of the language itself or it may perform actions that are available inan environment that the agent is connected to. Environment actions give theagent some control over the changes that occur in that environment. The typesof rules that are used in APLs varies. Generally speaking, however, rules have acondition that is evaluated on the agent’s mental state (rule head) and have acorresponding action or plan that is instantiated if the rule fired (rule body).Rules are evaluated and applied in a reasoning cycle that is part of the agentprogram’s interpreter. Agent interpreters for APLs implement a sense-plan-actcycle or a variant thereof. In a typical interpreter, for example, the percepts receivedfrom an environment are processed by some predefined mechanism. Mostoften this is an automatic mechanism (that may be customisable as in Jason) [4],but not necessarily; in Goal, for example, so-called percept rules are availablefor processing incoming percepts and similar rules are available for processingmessages received from other agents (2APL has similar rules [8]). During thisstage, either before or after processing the percepts, typically the messages receivedfrom other agents are processed. These steps are usually performed firstto ensure the mental state of the agent is up to date. Thereafter, the interpreterwill evaluate rules against the updated mental state and select applicable rules.After determining which rules are applicable one or more of these rules is fired,resulting in one or more options to perform an action or add a plan to a planbase. Some selection mechanism (that again may be customised as e.g. in Jason)then is used to arbitrate between these multiple options, or, as is the case in forexample Goal, a choice is randomly made. Finally, an action is executed eitherinternally or an action is sent to the environment for execution.One aspect of these interpreters is that they may generate multiple applicablerules and options for performing actions. If multiple options are available, thenthe agent program is underspecified in the sense that it does not determine aunique choice of action. It is this feature of agent architectures that we willexploit and can be used by a learning mechanism for optimising the agent’schoice of action [17, 18].2.2 Reinforcement LearningReinforcement learning [12] is a formal framework for optimally solving multistagedecision problems in environments where outcomes are only partly attributedto decision making by the agent and are partly stochastic. The generalidea is to describe the value of a decision problem at a given time step in termsof the payoffs received from choices made so far, and the value of the remainingproblem that results from those initial choices. Formally, at each time step t ina multistep problem, the agent perceives state s ∈ S of the environment and150