Combining Pattern Classifiers

xiv
PREFACE
learning communities, albeit scattered across various literature sources and disguised
under different names and notations. Yet some of the methods and algorithms
in the book are less well known. My choice was guided by how intuitive, simple, and
effective the methods are. I have tried to give sufficient detail so that the methods can
be reproduced from the text. For some of them, simple Matlab code is given as well.
The code is not foolproof nor is it optimized for time or other efficiency criteria. Its
sole purpose is to enable the reader to experiment. Matlab was seen as a suitable
language for such illustrations because it often looks like executable pseudocode.
I have refrained from making strong recommendations about the methods and
algorithms. The computational examples given in the book, with real or artificial
data, should not be regarded as a guide for preferring one method to another. The
examples are meant to illustrate how the methods work. Making an extensive experimental
comparison is beyond the scope of this book. Besides, the fairness of such a
comparison rests on the conscientiousness of the designer of the experiment. J.A.
Anderson says it beautifully [89]
There appears to be imbalance in the amount of polish allowed for the techniques.
There is a world of difference between “a poor thing – but my own” and “a poor
thing but his”.
The book is organized as follows. Chapter 1 gives a didactic introduction into the
main concepts in pattern recognition, Bayes decision theory, and experimental comparison
of classifiers. Chapter 2 contains methods and algorithms for designing the
individual classifiers, called the base classifiers, to be used later as an ensemble.
Chapter 3 discusses some philosophical questions in combining classifiers including:
“Why should we combine classifiers?” and “How do we train the ensemble?”
Being a quickly growing area, combining classifiers is difficult to put into unified
terminology, taxonomy, or a set of notations. New methods appear that have to
be accommodated within the structure. This makes it look like patchwork rather
than a tidy hierarchy. Chapters 4 and 5 summarize the classifier fusion methods
when the individual classifiers give label outputs or continuous-value outputs.
Chapter 6 is a brief summary of a different approach to combining classifiers termed
classifier selection. The two most successful methods for building classifier ensembles,
bagging and boosting, are explained in Chapter 7. A compilation of topics is
presented in Chapter 8. We talk about feature selection for the ensemble, error-correcting
output codes (ECOC), and clustering ensembles. Theoretical results found in
the literature are presented in Chapter 9. Although the chapter lacks coherence, it
was considered appropriate to put together a list of such results along with the details
of their derivation. The need of a general theory that underpins classifier combination
has been acknowledged regularly, but such a theory does not exist as yet.
The collection of results in Chapter 9 can be regarded as a set of jigsaw pieces awaiting
further work. Diversity in classifier combination is a controversial issue. It is a
necessary component of a classifier ensemble and yet its role in the ensemble performance
is ambiguous. Little has been achieved by measuring diversity and