Combining Pattern Classifiers

34 FUNDAMENTALS OF PATTERN RECOGNITION
By design, the classification regions of D correspond to the true highest posterior
probabilities. The bullet on the x-axis in Figure 1.12 splits R into R 1 (to the left) and
R 2 (to the right). According to Eq. (1.54), the Bayes error will be the area under
P(v 2 )p(xjv 2 )inR 1 plus the area under P(v 1)p(xjv 1 )inR 2 . The total area corresponding
to the Bayes error is marked in light gray. If the boundary is shifted to
the left or right, additional error will be incurred. We can think of this boundary
as the result from classifier D, which is an imperfect approximation of D . The
shifted boundary, depicted by an open circle, is called in this example the “real”
boundary. Region R 1 is therefore R 1 extended to the right. The error calculated
through Eq. (1.54) is the area under P(v 2 )p(xjv 2 ) in the whole of R 1 , and extra
error will be incurred, measured by the area shaded in dark gray. Therefore, using
the true posterior probabilities or an equivalent set of discriminant functions guarantees
the smallest possible error rate, called the Bayes error.
Since the true probabilities are never available in practice, it is impossible to calculate
the exact Bayes error or design the perfect Bayes classifier. Even if the probabilities
were given, it will be difficult to find the classification regions in R n and
calculate the integrals. Therefore, we rely on estimates of the error as discussed
in Section 1.3.
1.5.6 Multinomial Selection Procedure for Comparing Classifiers
Alsing et al. [26] propose a different view of classification performance. The classifiers
are compared on a labeled data set, relative to each other in order to identify
which classifier has most often been closest to the true class label. We assume
that each classifier gives at its output a set of c posterior probabilities, one for
each class, guessing the chance of that class being the true label for the input vector
x. Since we use labeled data, the posterior probabilities for the correct label of x are
sorted and the classifier with the largest probability is nominated as the winner for
this x.
Suppose we have classifiers D 1 , ..., D L to be compared on a data set Z of size N.
The multinomial selection procedure consists of the following steps.
1. For i ¼ 1, ..., c,
(a) Use only the N i data points whose true label is v i . Initialize an N i L performance
array T.
(b) For every point z j , such that l(z j ) ¼ v i , find the estimates of the posterior
probability P(v i jz j ) guessed by each classifier. Identify the largest posterior
probability, store a value of 1 for the winning classifier D q by setting
T( j, q) ¼ 1 and values 0 for the remaining L 1 classifiers,
T( j, k) ¼ 0, k ¼ 1, ..., L, k = q.
(c) Calculate an estimate of each classifier being the winner for class v i
assuming that the number of winnings follows a binomial distribution.
The estimate of this probability will be the total number of 1s stored