I__. - International Military Testing Association

classification tables and CA's were also computed for each
set of weights. The CA's using weights from the cross-validation
samples ranged from 70% (908X0) to 92% (121X0); and for the
validation samples, from 68% (231X0) to 92% (121X0). Of the
4,104 tasks classified within the 26 AFSs, only four "D" tasks
were classified as "A" tasks and only four @'A" tasks were
classified as @rD@t tasks. Although it is desirable to have zero A
to D or D to A misclassifications (because the test development
team is being advised incorrectly to write or not write an item),
infrequent misclassifications of this type should not adversely
affect the construction of a valid SKT. The team can rectify
these discrepancies with the permission of the group facilitator. *:
CA's computed for the combined data ranged from 71% (908X0) to
90% (112X0). In general, the predictive accuracies using
combined samples were higher than those for the validation
samples referred to earlier, but all differences were small (less
than 6%). Only two @ID" tasks were classified as rcA't tasks and
two *'Aft tasks were classified as "D" tasks. Squared and
interactive predictor terms were added to the model for each AFS
in an attempt to increase classification accuracy, but only small
increases in accuracy were observed. In fact, for some AFSs,
classification accuracy decreased.
What is adequate classification accuracy in the context of
generating an ATO? The table having the lowest CA value (68%) is
shown in Figure 2. It was generated by applying the 112X0
weights from the validation sample to the cross-validation
sample. The impact of the misclassifications in the table is
probably not too severe when it is recalled that the AT0 is a
guide for SMEs to use in developing an SXT, and they are free to
select tasks from any of the importance categories within the
restrictions delineated above.
Pfedl,otd CtuJfloation
Flgwo 2. Claultlcatlon Tsbk wlth Lorn.1 a Valu.
The r's for the combined data for each AFS ranged from .51
(908X0) to .91 (112X0). A hierarchical clustering of the
regression equations for ail 26 AFSs showed small decreases in r
throughout most of the clustering process. For example, the
overall r dropped from .84 at the 26-group stage (i.e., a
separate regression equation for each of the 26 AFSs) to .79 at
the 5-group stage. Thereafter, the drops in r to the l-group
Stage were .02, .02, .04, and .12, respectively.
The gradual drop in r's until the clustering at the l-group
stage makes identification of an "optimal clustering stage"
difficult. Therefore, classification was also examined at
various stages. The CA's of equations at the l-group stage
314