NUI Galway – UL Alliance First Annual ENGINEERING AND - ARAN ...

Applications of Reinforcement Learning in Gaming Domains
Frank G. Glavin Michael G. Madden
College of Engineering and Informatics, National University of Ireland, Galway
Frank.Glavin@gmail.com, Michael.Madden@nuigalway.ie
Abstract
This paper introduces the concept of Reinforcement
Learning (RL) and then describes the elements of a
reinforcement learning system. Some related work is
briefly mentioned and then some potentially relevant
domains are discussed. The paper is concluded by
stating the goal of this research.
1. Introduction
Reinforcement learning is a branch of Artificial
Intelligence in which a learner, often called an agent,
interacts with an environment in order to achieve an
explicit goal. The agent receives feedback for its actions
in the form of numerical rewards. The agent learns from
its interactions with the environment and aims to
maximize the reward values that it receives over time.
The agent must make a tradeoff between exploring the
effects of taking novel actions and exploiting the
knowledge that has been acquired from earlier
exploration.
2. Reinforcement Learning System
In addition to the agent and its environment, Sutton
and Barto [1] have identified four primary sub-elements
that form a reinforcement learning system. These are: a
policy; a reward function; a value function; and a model
of the environment. A policy is a definition of the
proposed agent’s behaviour in a given situation. This is
essentially a mapping from states to actions. The reward
function assigns a single numeric reward value to each
state in the environment to represent the desirability of
being in the state. These values can be used as the basis
of altering the agent’s policy. The value function
estimates the amount of reward that an agent can expect
to acquire from the current state over possible future
states. The values are estimated with a view to increase
the amount of rewards achieved over time. A model
consists of the agents internal representation of the
environment and is used to predict future states and
rewards before they are actually experienced, which is
useful for planning ahead.
3. Related Work
Gaming environments have been widely used as test
beds for reinforcement learning algorithms. One of the
most successful applications was Gerald Tesauro’s TD-
Gammon [2] which was developed in the early 1990s.
This used an Artificial Neural Network which was
trained using a temporal difference learning algorithm
called TD-Lamda[3]. It achieved a level of play close to
the top human players in the world.
9
4. Potentially Relevant Domains
4.1. RoboCup Soccer Tournament
The objective of this annually held tournament is to
promote research into robotics and Artificial
Intelligence. Teams of researchers from around the
world compete every year in both the robotic and
software simulation competitions. The overall goal is to
produce a team of fully autonomous humanoid robots
that can play against, and beat, the current world cup
holders of 2050[4].
4.2. First Person Shooter (FPS) Bots
As graphics in modern computer games move closer
to photorealism, the emphasis is switching to improving
in-game artificial intelligence. Rule-based and
traditional scripting systems are being replaced by
intelligent reinforcement learning agents. There has
been some recent promising work in this area but there
is plenty of scope for improvement.
4.3. Educational and Training Software
This would involve creating an intelligent agent that
could build up a user profile based on what it learns
from the user’s interactions. This could possibly be
applied to “brain training”, “typing tutor” or similar
type games. The practicability of such an approach
would have to be tested as there is very little reported
work of reinforcement learning being applied to such
problems in the literature.
5. Conclusion
The overall goal of this research will be to examine the
current state of the art in the application of
Reinforcement Learning to gaming domains. Future
work will involve identifying a specific gaming domain
following an extensive literature review. Novel
experimentation will then be carried out in this area.
6. References
[1] Sutton, R. S., & A. G Barto, and E.F. Roberts,
Reinforcement Learning: An Introduction, MIT Press,
Cambridge, MA, 1998.
[2] Tesauro, G., “TD-Gammon: A self-teaching backgammon
program achieves master-level play.” Neural
Computation, 6(2), 215-219, 1995.
[3] Sutton, R. S., “Learning to predict by the methods of
temporal differences” Machine Learning, 3, 9-44.
1988.
[4] Http://www.robocup.org