Applied Statistics Using SPSS, STATISTICA, MATLAB and R

258 6 Statistical Classification
Statistical software products such as SPSS and STATISTICA allow the
selection of the cases used for training and for testing linear discriminant
classifiers. With SPSS, it is possible to use a selection variable, easing the task of
specifying randomly selected samples. SPSS also affords performing a leave-oneout
classification. With STATISTICA, one can initially select the cases used for
training (Selection Conditions option in the Tools menu), and once the
classifier is designed, specify test cases (Select Cases button in the
Classification tab of the command window). In MATLAB and R one may
create a case-selecting vector, called a filter, with random 0s and 1s.
Example 6.14
Q: Consider the two-class cork-stopper classifier, with two features, presented in
section 6.2.2 (see classification matrix in Table 6.3). Evaluate the performance of
this classifier using the partition method with k = 3, and the leave-one-out method.
A: Using the partition method with k = 3, a test set estimate of Pet = 9.9 % was
obtained, which is near the training set error estimate of 10%. The leave-one-out
method also produces Pet = 10 % (see Table 6.11; the “Original” matrix is the
training set estimate, the “Cross-validated” matrix is the test set estimate). The
closeness of these figures is an indication of reliable error estimation for this high
dimensionality ratio classification problem (n/d = 25). Using formula 6.28 the 95%
confidence limits for these error estimates are: s = 0.03 ⇒ Pe = 10% ± 5.9%.
Table 6.11. Listing of the classification matrices obtained with SPSS, using the
leave-one-out method in the classification of the first two classes of the cork-
stopper data with two features.
Predicted Group Membership Total
C 1 2
Original Count 1 49 1 50
2 9 41 50
% 1 98.0 2.0 100
2 18.0 82.0 100
Cross-validated Count 1 49 1 50
2 9 41 50
% 1 98.0 2.0 100
2 18.0 82.0 100
Example 6.15
Q: Consider the three-class, cork-stopper classifier, with four features, determined
in Example 6.13. Evaluate the performance of this classifier using the leave-oneout
method.