Combining Pattern Classifiers

12 FUNDAMENTALS OF PATTERN RECOGNITION
l ij denoting the loss incurred by assigning label v i , given that the true label of the
object is v j . If the classifier is unsure about the label, it may refuse to make a
decision. An extra class (called refuse-to-decide) denoted v cþ1 can be added to
V. Choosing v cþ1 should be less costly than choosing a wrong class. For a problem
with c original classes and a refuse option, the loss matrix is of size (c þ 1) c. Loss
matrices are usually specified by the user. A zero–one (0–1) loss matrix is defined
as l ij ¼ 0 for i ¼ j and l ij ¼ 1 for i = j, that is, all errors are equally costly.
1.4 EXPERIMENTAL COMPARISON OF CLASSIFIERS
There is no single “best” classifier. Classifiers applied to different problems and
trained by different algorithms perform differently [15–17]. Comparative studies
are usually based on extensive experiments using a number of simulated and real
data sets. Dietterich [14] details four important sources of variation that have to
be taken into account when comparing classifier models.
1. The choice of the testing set. Different testing sets may rank differently classifiers
that otherwise have the same accuracy across the whole population.
Therefore it is dangerous to draw conclusions from a single testing experiment,
especially when the data size is small.
2. The choice of the training set. Some classifier models are called instable [18]
because small changes in the training set can cause substantial changes of the
classifier trained on this set. Examples of instable classifiers are decision tree
classifiers and some neural networks. (Note, all classifier models mentioned
will be discussed later.) Instable classifiers are versatile models that are
capable of adapting, so that all training examples are correctly classified.
The instability of such classifiers is in a way the pay-off for their versatility.
As we shall see later, instable classifiers play a major role in classifier
ensembles. Here we note that the variability with respect to the training
data has to be accounted for.
3. The internal randomness of the training algorithm. Some training algorithms
have a random component. This might be the initialization of the parameters
of the classifier, which are then fine-tuned (e.g., backpropagation algorithm
for training neural networks), or a random-based procedure for tuning the classifier
(e.g., a genetic algorithm). Thus the trained classifier might be different
for the same training set and even for the same initialization of the parameters.
4. The random classification error. Dietterich [14] considers the possibility of
having mislabeled objects in the testing data as the fourth source of variability.
The above list suggests that multiple training and testing sets should be used, and
multiple training runs should be carried out for classifiers whose training has a
stochastic element.