Combining Pattern Classifiers

16 FUNDAMENTALS OF PATTERN RECOGNITION
counts needed for the calculation of x 2 in Eq. (1.11).
From Eq. (1.11)
N 11 ¼ 80 þ 2 ¼ 82 N 10 ¼ 0 þ 2 ¼ 2
N 01 ¼ 9 þ 1 ¼ 10 N 00 ¼ 3 þ 3 ¼ 6
x 2 ¼
(j10 2j 1)2
10 þ 2
¼ 49 4:0833 (1:14)
12
Since the calculated x 2 is greater than the tabulated value of 3.841, we reject the
null hypothesis and accept that LDC and 9-nn are significantly different. Applying
the difference of proportions test to the same pair of classifiers gives p ¼
(0:84 þ 0:92)=2 ¼ 0:88, and
0:84 0:92
z ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
1:7408 (1:15)
(2 0:88 0:12)=(100) In this case jzj is smaller than the tabulated value of 1.96, so we cannot reject the null
hypothesis and claim that LDC and 9-nn have significantly different accuracies.
Which of the two decisions do we trust? The McNemar test takes into account the
fact that the same testing set Z ts was used whereas the difference of proportions does
not. Therefore, we can accept the decision of the McNemar test. (Would it not have
been better if the two tests agreed?)
1.4.2 Cochran’s Q Test and F-Test
To compare L . 2 classifiers on the same testing data, the Cochran’s Q test or the
F-test can be used.
1.4.2.1 Cochran’s Q Test. Cochran’s Q test is proposed for measuring whether
there are significant differences in L proportions measured on the same data [20].
This test is used in Ref. [21] in the context of comparing classifier accuracies. Let
p i denote the classification accuracy of classifier D i . We shall test the hypothesis
for no difference between the classification accuracies (equal proportions):
H 0 : p 1 ¼ p 2 ¼¼p L (1:16)
If there is no difference, then the following statistic is distributed approximately as
x 2 with L 1 degrees of freedom
Q C ¼ (L
1) L P L
i¼1 G2 i T 2
P
LT
Nts
j¼1 (L (1:17)
j) 2