Combining Pattern Classifiers

CONTENTS
vii
3 Multiple Classifier Systems 101
3.1 Philosophy / 101
3.1.1 Statistical / 102
3.1.2 Computational / 103
3.1.3 Representational / 103
3.2 Terminologies and Taxonomies / 104
3.2.1 Fusion and Selection / 106
3.2.2 Decision Optimization and Coverage Optimization / 106
3.2.3 Trainable and Nontrainable Ensembles / 107
3.3 To Train or Not to Train? / 107
3.3.1 Tips for Training the Ensemble / 107
3.3.2 Idea of Stacked Generalization / 109
3.4 Remarks / 109
4 Fusion of Label Outputs 111
4.1 Types of Classifier Outputs / 111
4.2 Majority Vote / 112
4.2.1 Democracy in Classifier Combination / 112
4.2.2 Limits on the Majority Vote Accuracy: An Example / 116
4.2.3 Patterns of Success and Failure / 117
4.3 Weighted Majority Vote / 123
4.4 Naive Bayes Combination / 126
4.5 Multinomial Methods / 128
4.5.1 Behavior Knowledge Space Method / 128
4.5.2 Wernecke’s Method / 129
4.6 Probabilistic Approximation / 131
4.6.1 Calculation of the Probability Estimates / 134
4.6.2 Construction of the Tree / 135
4.7 Classifier Combination Using Singular Value Decomposition / 140
4.8 Conclusions / 144
Appendix 4A Matan’s Proof for the Limits on the Majority
Vote Accuracy / 146
Appendix 4B Probabilistic Approximation of the
Joint pmf for Class-Label Outputs / 148
5 Fusion of Continuous-Valued Outputs 151
5.1 How Do We Get Probability Outputs? / 152
5.1.1 Probabilities Based on Discriminant Scores / 152