Evolutionary Computation : A Unified Approach

108 CHAPTER 5. EVOLUTIONARY ALGORITHMS AS PROBLEM SOLVERS
of models of sufficient descriptive power are generally so large as to prohibit any form of
systematic search. However, as we discussed in the previous section, EAs can often be used
to search large complex spaces effectively, including the model spaces of interest here.
Consequently, ML-EAs tend to adopt a more top-down approach of searching a model
space by means of a population of models that compete with each other, not unlike scientific
theories in the natural sciences. In order for this to be effective, a notion of model “fitness”
must be provided in order to bias the search process in a useful way. The first thought one
might have is to define the fitness of a model in terms of its observed performance on the
provided training examples. However, just as we see in other approaches, ML-EAs with no
other feedback or bias will tend to overfit the training data at the expense of generality.
So, for example, if an ML-EA is attempting to learn a set of classification rules using only
training set performance as the measure of fitness, it is not at all unusual to see near-perfect
rule sets emerge that consist of approximately one rule per training example!
Stated another way, for a given set of training examples, there are a large number of
theories (models) that have identical performance on the training data, but can have quite
different predictive power. Since, by definition, we cannot learn from unseen examples, this
generality must be achieved by other means. A standard technique for doing so is to adopt
some sort of “Occam’s razor” approach: all other things being equal, select a simpler model
over a more complex one.
For ML-EAs this typically is achieved by augmenting the fitness function to include both
performance of the training data and the parsimony of the model. Precise measurements
of parsimony can be quite difficult to define in general. However, rough estimates based on
the size of a model measured in terms of its basic building blocks have been shown to be
surprisingly effective. For example, using the number of rules in a rule set or the number of
hidden nodes in an artificial neural network as an estimate of parsimony works quite well.
Somewhat more difficult from an ML-EA designer’s point of view is finding an appropriate
balance between parsimony and performance. If one puts too much weight on generality,
performance will suffer and vice versa. As we will see in a later section in this chapter, how
best to apply EAs to problems involving multiple conflicting objectives is a challenging
problem in general. To keep things simple, most ML-EAs adopt a fairly direct approach
such as:
fitness(model) =performance(model) − w ∗ parsimony(model)
where the weight w is empirically chosen to discourage overfitting, and parsimony is a simple
linear or quadratic function of model size (Smith, 1983; Bassett and De Jong, 2000).
Just as was the case in the previous sections, applying EAs to machine-learning problems
is not some sort of magic wand that renders existing ML techniques obsolete. Rather, ML-
EAs complement existing approaches in a number of useful ways:
• Many of the existing ML techniques are designed to learn “one-shot” classification
tasks, in which problems are presented as precomputed feature vectors to be classified
as belonging to a fixed set of categories. When ML-EAs are applied to such problems,
they tend to converge more slowly than traditional ML techniques but often to more
parsimonious models (De Jong, 1988).