Combining Pattern Classifiers

24 FUNDAMENTALS OF PATTERN RECOGNITION
TABLE 1.7 Comparison of Testing Errors of Two Classifiers.
e 1 e 2 z jzj . 1.96? Outcome
Experiment 1 21.37 22.00 22.11 Yes Different (e 1 , e 2 )
Experiment 2 23.38 22.00 4.54 Yes Different (e 1 . e 2 )
classifier has a testing error of 21.37 percent (the minimum in the bottom plot). The
mistake is in that the testing set was used to find the best value of r.
Let us use the difference of proportions test for the errors of the classifiers. The
testing error of our quadratic classifier (QDC) at the final 20th step (corresponding to
the minimum training error) is 23.38 percent. Assume that the competing classifier
has a testing error on this data set of 22.00 percent. Table 1.7 summarizes the results
from two experiments. Experiment 1 compares the best testing error found for QDC,
21.37 percent, with the rival classifier’s error of 22.00 percent. Experiment 2 compares
the end error of 23.38 percent (corresponding to the minimum training error of
QDC), with the 22.00 percent error. The testing data size in both experiments is
N ts ¼ 19,000.
The results suggest that we would decide differently if we took the best testing
error rather than the testing error corresponding to the best training error. Experiment
2 is the fair comparison in this case.
A point raised by Duin [16] is that the performance of a classifier depends upon
the expertise and the willingness of the designer. There is not much to be done for
classifiers with fixed structures and training procedures (called “automatic” classifiers
in Ref. [16]). For classifiers with many training parameters, however, we can
make them work or fail due to designer choices. Keeping in mind that there are
no rules defining a fair comparison of classifiers, here are a few (non-Candide’s)
guidelines:
1. Pick the training procedures in advance and keep them fixed during training.
When publishing, give enough detail so that the experiment is reproducible by
other researchers.
2. Compare modified versions of classifiers with the original (nonmodified) classifier.
For example, a distance-based modification of k-nearest neighbors
(k-nn) should be compared with the standard k-nn first, and then with other
classifier models, for example, neural networks. If a slight modification of a
certain model is being compared with a totally different classifier, then it is
not clear who deserves the credit, the modification or the original model itself.
3. Make sure that all the information about the data is utilized by all classifiers to
the largest extent possible. For example, a clever initialization of a prototypebased
classifier such as the learning vector quantization (LVQ) can make it
favorite among a group of equivalent but randomly initialized prototype classifiers.