Combining Pattern Classifiers

20 FUNDAMENTALS OF PATTERN RECOGNITION
and
P (2) ¼ P (2)
A P (2)
B
The estimated mean and variance of the differences, for this two-fold cross-validation
run, are calculated as
P ¼ P(1) þ P (2)
; s 2 ¼ P (1) P 2
þ P
(2) P 2
2
(1:26)
Let P (1)
i denote the difference P (1) in the ith run, and s 2 i denote the estimated variance
for run i, i ¼ 1, ..., 5. The proposed ~t statistic is
P (1)
1
~t ¼ q ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
(1=5) P (1:27)
5
i¼1 s2 i
Note that only one of the ten differences that we will calculate throughout this experiment
is used in the numerator of the formula. It is shown in Ref. [14] that under the
null hypothesis, ~t has approximately a t distribution with five degrees of freedom.
Example: Comparison of Two Classifier Models Through Cross-Validation
Tests. The banana data set used in the previous examples is suitable for experimenting
here because we can generate as many as necessary independent data sets from
the same distribution. We chose the 9-nn and Parzen classifiers. The Matlab code for
the three cross-validation methods discussed above is given in Appendices 1A to 1C
at the end of this chapter. PRTOOLS toolbox for Matlab, version 2 [19], was used to
train and test the two classifiers.
K-Hold-Out Paired t-Test. The training and testing data sets used in the previous
example were pooled and the K-hold-out paired t-test was run with K ¼ 30, as
explained above. We chose to divide the data set into halves instead of a 2=3 to
1=3 split. The test statistic (1.25) was found to be t ¼ 1:9796. At level of significance
0.05, and degrees of freedom K 1 ¼ 29, the tabulated value is 2.045
(two-tailed test). Since the calculated value is smaller than the tabulated value,
we cannot reject the null hypothesis. This test suggests that 9-nn and Parzen classifiers
do not differ in accuracy on the banana data. The averaged accuracies over the
30 runs were 92.5 percent for 9-nn, and 91.83 percent for Parzen.
K-Fold Cross-Validation Paired t-Test. We ran a 10-fold cross-validation for the
set of 200 data points, so each testing set consisted of 20 objects. The ten testing
accuracies for 9-nn and Parzen are shown in Table 1.5.
From Eq. (1.25) we found t ¼ 1:0000. At level of significance 0.05, and degrees
of freedom K 1 ¼ 9, the tabulated value is 2.262 (two-tailed test). Again, since the