An Introduction to Genetic Algorithms - Boente

Figure 3.6: A schematic illustration of the components of an agent in ERL. The agent's genotype is a bit string
that encodes the weights of two neural networks: an evaluation network that maps the agent's current state to
an evaluation of that state, and an action network that maps the agent's current state to an action to be taken at
the next time step. The weights on the evaluation network are constant during an agent's lifetime but the
weights on the action network can be modified via a reinforcement learning method that takes its positive or
negative signal from the evaluation network. (The networks displayed here are simplified for clarity.) The
agent's genotype is not modified by this learning procedure, and only the genotype is passed from an agent to
its offspring. (Reprinted from Christopher G. Langton et al., eds., Artificial Life: Volume II; © 1992
Addison−Wesley Publishing Company, Inc. Reprinted by permission of the publisher.)
Evolutionary Reinforcement Learning
Chapter 3: Genetic Algorithms in Scientific Models
A second computational demonstration of the Baldwin effect was given by David Ackley and Michael
Littman (1992). Their primary goal was to incorporate "reinforcement learning" (an unsupervised learning
method) into an evolutionary framework and to see whether evolution could produce individuals that not only
behaved appropriately but also could correctly evaluate the situations they encountered as beneficial or
dangerous for future survival. In Ackley and Littman's Evolutionary Reinforcement Learning (ERL) model,
individuals ("agents") move randomly on a finite two−dimensional lattice, encountering food, predators,
hiding places, and other types of entities. Each agent's "state" includes the entities in its visual range, the level
of its internal energy store, and other parameters.
The components making up an individual are illustrated schematically in figure 3.6. Each agent possesses two
feedforward neural networks: an evaluation network that takes as input the agent's state at time t and produces
on its single output unit an activation representing a judgment about how good that state is for the agent, and
an action network that takes as input the agent's state at time t and produces on its two output units a code for
the action the agent is to take on that time step. The only possible actions are moves from the current lattice
site to one of the four neighboring sites, but actions can result in eating, being eaten, and other less radical
consequences. The architectures of these two networks are common to all agents, but the weights on the links
can vary between agents. The weights on a given agent's evaluation network are fixed from birth—this
network represents innate goals and desires inherited from the agent's ancestors (e.g., "being near food is
good"). The weights on the action network change over the agent's lifetime according to a
reinforcementlearning algorithm that is a combination of back−propagation and standard reinforcement
learning.
An agent's genome is a bit string encoding the permanent weights for the evaluation network and the initial
weights for the action network. The network architectures are such that there are 84 possible weights, each
encoded by four bits. The length of a chromosome is thus 336 bits.
Agents have an internal energy store (represented by a real number) which must be kept above a certain level
to prevent death; this is accomplished by eating food that is encountered as the agent moves from site to site
on the lattice. An agent must also avoid predators, or it will be killed. An agent can reproduce once it has
72