I__. - International Military Testing Association

Air Force enlisted specialties. The first objective was to determine the
extent to which SME ratings on the CVR forms impact subsequent identification
of items as acceptable or unacceptable for reuse. The second is to determine
how SMEs and project psychologists perceive the value and usefulness of the
forms. Ninety-four SMEs, representing 25 AFSs, assigned to USAFOMS for SKT
rewrite duties were asked to rate test items from their respective E-5 and
E-6/7 grade-level SKTs using the CVR forms. USAFOMS test development proce-
dures requires the completion of this step prior to the SMEs’ designation of
an item as either acceptable for continued use on subsequent SKTs or as unac-
ceptable for reuse. Once again, these test items were rated using Lawshe’s
3-point scale. A rating of 2 was given to items whose content was essential.
A rating of 1 was assigned to those items whose content was useful, but not
essential, and a rating of 0 was assigned to those items whose content was
not necessary for successful performance in the AFS. In all, 19,700 ra~tings .
were obtained from 94 raters for 2 SKT levels (E-5 and E-617).
Results
Intraclass correlations for each of the 25 AFSs were computed to determine
the interrater reliabilities for the group of SMEs from each specialty. All
but two of the calculated values had p < .05. (The higher reliability values
obtained seemed to be associated with the more technologically specialized
fields where there is little room for variance of procedures across the
Air Force, thus leading to more agreement among SMEs on items which test essential
knowledge. Lower values seemed to be associated with broader specialties
where there is more variance in day-to-day jobs performed and hence,
less agreement and lower values of reliability.)
The average CVR for items chosen as acceptable and for those designated for
deactivation were also calculated for each test project. The mean CVR value
for all deactivated items was 1.28 and the mean CVR value for all acceptable
items was 1.43. These results conformed to our expectations that on the
whole, items selected for deactivation would have lower content validity ratings
than those chosen as acceptable. The average CVR value for all 19,700
ratings obtained was 1.40. This average reflects the fact that on the whole,
Air Force SKTs are viewed as being relatively high in content validity.
To determine the actual impact, if any, of the content validity ratings on
the subsequent identification of an item for reuse, a chi-square test of statistical
significance was computed. The null hypothesis (H 1 for this test
states that there is no difference between the proportion o? items selected
as acceptable and unacceptable in each rating category. The alternative hypothesis
(H 1 is that the distribution of items in each rating category differs
from t%e hypothesized one. The results, as shown in Table 1, indicate
significant differences between expected and observed values for acceptable
and deactivated items in each rating category, with the largest differences
occurring between ratings of 2 and 0. As shown, 203 more ratings of 0 were
observed for deactivated items than was expected, while 234 more ratings of 2
were observed for acceptable items than was expected. A chi-square value of
199.7 (df=2) was obtained, indicating significance at the .Ol level. On
this basis, the null hypothesis was rejected, indicating a disproportional
representation of items selected as acceptable and unacceptable in each rating
category. This shows that item content validity did impact subsequent
identification of item acceptability.
A point-biserial correlation coefficient relating identification of an item
as either acceptable or deactivated with the item's average content validity
237