I__. - International Military Testing Association

ating was also computed. The resulting correlation coefficient, .0894, is
significant at the .Ol level, yet is rather low. This can be attributed to
the fact that of the 19,700 total ratings, 8,387 were ratings of 1. Since
the largest differences in proportional representations were found to be in
the rating categories of 0 and 2 and the number of ratings of 1 were within
31 ratings of the expected value, it appears that the correlation coefficient
may have been decreased by the large number of ratings of 1.
Table 1. Comparison of Acceptable and Deactivated Test Items
OBSERVED
Acceptable Deactivated
Rating 0 1213 515 1728
Rating 1 6840 1547 8387
Rating 2 8086 1499 9585
16139 3561 19700
-.-
EXPECTED
Acceptable Deactivated
Rating 0 1415.6 312.4 1728
Rating 1 6871 1516 8387
Rating 2 7852.4 1732.6 9585
16139 3561 17700
X2 = 199.7 df = 2 PC.01
Through the analysis of the data, a number of unusual cases surfaced. The
data contained in these cases was contrary to expectations and evoked further
analysis . For instance, 10 of the 5,200 test items evaluated were given rat-
ings of 0 (not necessary) by all SMEs, yet still retained as acceptable
items. Perhaps the SMEs reconsidered the item content and found it essential
to successful job performance. Furthermore, 117 of the 940 deactivated items
were given ratings of 2 (essential) by all SMEs. After obtaining the reasons
for deactivation, these items were broken down into six categories:
REASON FOR DEACTIVATION # ITEMS % OF TOTAL
Poor Statistics/No Acceptable Revision
Inadequate Reference
Obsolete I tern
Two or More (or No) Correct Answers
Low Content Validity
Inadvertent Duplication of Item
57 49%
37 32%
11 9%
8 7%
3 3%
1 1 .!
As stated earlier, there are other requirements not directly related to content
validity that must be met if an item is to be retained. These include
238