An Introduction to Genetic Algorithms - Boente

initial pattern on the input units. This would be trivial, except that the number of hidden units is smaller than
the number of input units, so some encoding and decoding must be done.
These are all relatively easy problems for multi−layer neural networks to learn to solve under
back−propagation. The networks had different numbers of units for different tasks (ranging from 5 units for
the XOR task to 20 units for the encoder/decoder task); the goal was to see if the GA could discover a good
connection topology for each task. For each run the population size was 50, the crossover rate was 0.6, and the
mutation rate was 0.005. In all three tasks, the GA was easily able to find networks that readily learned to map
inputs to outputs over the training set with little error. However, the three tasks were too easy to be a rigorous
test of this method—it remains to be seen if this method can scale up to more complex tasks that require much
larger networks with many more interconnections. I chose the project of Miller, Todd, and Hegde to illustrate
this approach because of its simplicity. For several examples of more sophisticated approaches to evolving
network architectures using direct encoding, see Whitley and Schaffer 1992.
Grammatical Encoding
Chapter 2: Genetic Algorithms in Problem Solving
The method of grammatical encoding can be illustrated by the work of Hiroaki Kitano (1990), who points out
that direct−encoding approachs become increasingly difficult to use as the size of the desired network
increases. As the network's size grows, the size of the required chromosome increases quickly, which leads to
problems both in performance (how high a fitness can be obtained) and in efficiency (how long it takes to
obtain high fitness). In addition, since direct−encoding methods explicitly represent each connection in the
network, repeated or nested structures cannot be represented efficiently, even though these are common for
some problems.
The solution pursued by Kitano and others is to encode networks as grammars; the GA evolves the grammars,
but the fitness is tested only after a "development" step in which a network develops from the grammar. That
is, the "genotype" is a grammar, and the "phenotype" is a network derived from that grammar.
A grammar is a set of rules that can be applied to produce a set of structures (e.g., sentences in a natural
language, programs in a computer language, neural network architectures). A simple example is the following
grammar:
Here S is the start symbol and a nonterminal, a and b are terminals, and µ is the empty−string terminal.(S ’ µ
means that S can be replaced by the empty string.) To construct a structure from this grammar, start with S,
and replace it by one of the allowed replacements given by the righthand sides (e.g., S ’ aSb). Now take the
resulting structure and replace any nonterminal (here S) by one of its allowed replacements (e.g., aSb ’
aaSbb). Continue in this way until no nonterminals are left (e.g., aaSbb ’ aabb, using S ’ µ). It can easily be
shown that the set of structures that can be produced by this grammar are exactly the strings a n b n consisting of
the same number of as and bs with all the as on the left and all the bs on the right.
Kitano applied this general idea to the development of neural networks using a type of grammar called a
"graph−generation grammar," a simple example of which is given in figure 2.22a Here the right−hand side of
each rule is a 2 × 2 matrix rather than a one−dimensional string. Capital letters are nonterminals, and
lower−case letters are terminals. Each lower−case letter from a through p represents one of the 16 possible 2 ×
2 arrays of ones and zeros. In contrast to the grammar fora n b n given above, each nonterminal in this particular
grammar has exactly one right−hand side, so there is only one structure that can be formed from this
grammar: the 8 x 8 matrix shown in figure 2.22b This matrix can be interpreted as a connection matrix for a
55