Combining Pattern Classifiers

22 FUNDAMENTALS OF PATTERN RECOGNITION
level of significance 0.05). Therefore we cannot conclude that there is a significant
difference between the two models on this data set.
It is intuitively clear that simple models or stable classifiers are less likely to be
overtrained than more sophisticated models. However, simple models might not be
versatile enough to fit complex classification boundaries. More complex models
(e.g., neural networks and prototype-based classifiers) have a better flexibility but
require more system resources and are prone to overtraining. What do “simple”
and “complex” mean in this context? The main aspects of complexity can be summarized
as [23]
. training time and training complexity;
. memory requirements (e.g., the number of the parameters of the classifier that
are needed for its operation); and
. running complexity.
1.4.4 Experiment Design
When talking about experiment design, I cannot refrain from quoting again and
again a masterpiece of advice by George Nagy titled “Candide’s practical principles
of experimental pattern recognition” [24]:
Comparison of Classification Accuracies
Comparisons against algorithms proposed by others are distasteful and should be
avoided. When this is not possible, the following Theorem of Ethical Data Selection
may prove useful. Theorem: There exists a set of data for which a candidate algorithm
is superior to any given rival algorithm. This set may be constructed by omitting from
the test set any pattern which is misclassified by the candidate algorithm.
Replication of Experiments
Since pattern recognition is a mature discipline, the replication of experiments on new
data by independent research groups, a fetish in the physical and biological sciences, is
unnecessary. Concentrate instead on the accumulation of novel, universally applicable
algorithms. Casey’s Caution: Do not ever make your experimental data available to
others; someone may find an obvious solution that you missed.
Albeit meant to be satirical, the above principles are surprisingly widespread and
closely followed! Speaking seriously now, the rest of this section gives some practical
tips and recommendations.
Example: Which Is the “Best” Result? Testing should be carried out on previously
unseen data. Let D(r) be a classifier with a parameter r such that varying
r leads to different training accuracies. To account for this variability, here we
use a randomly drawn 1000 objects from the Letter data set. The remaining