I__. - International Military Testing Association

clear reference support and acceptable item statistics, thus most of the 117
items were deactivated for valid reasons. It was surprising, however, that
3% of the items in question had "low content validity" cited on the i tern
record card as the reason for deactivation when originally all SMEs had felt
the item content was essential for successful job performance. These results
could be due to administrative error, SME reconsideration of the item con-
tent, or a number of other possible reasons.
The second objective of this research was to determine how SMEs and project
psychologists perceived the value and usefulness of the CVR forms. A three
question survey was administered to 21 project psychologists at USAFOMS. A
similar survey was administered to 151 SMEs upon completion of the rating
forms and selection of items for deactivation. A four-point rating scale was
used for the responses: strongly agree, agree, disagree, and strongly. dis-
agree. The questions and summary of responses are as follows:
(11 When selecting previously used items to be reused, the Content Validity
Rating forms hglped identify those items most essential to successful
performance in the specialty.
49% (74) SMEs answered positively (agree or strongly agree)
52% (111 project psychologists answered positively
(2) The Content Validity Rating forms helped bring out different points
of view for discussion.
56% (85) SMEs answered positively
76% (16) project psychologists answered positively
(3) The Content Validity Rating forms were a valuable tool in selecting
Discussion
items to be reused.
39% (59) SMEs answered positively
43% (9) project psychologists answered positively
Through the analysis previously described, it became apparent that there was
a significant impact of content validity ratings on subsequent identification
of an item as acceptable or deactivated. For instance, with the 25 projects
sampled, there were 52 individual SKTs examined. Of these 52 SKTs, 37 had
higher average CVR values for acceptable items than the average CVR values of
the deactivated items which is what would be expected -- a positive differ-
ence between the two. It is also important to note that 6 of the 52 SKTs are
not applicable in this analysis since in these cases all 100 items were designated
as acceptable and thus, there was no average CVR value for deactivated
items. The 9 remaining SKTs had higher average CVR values for the deactivated
items than the average CVR values for the acceptable items. Of these 9
SKTs, 2 were from projects with insignificant interrater reliabilities. Even
though the expected effect did not hold true for every case, overall, items
higher in content’ validity had a greater chance of being acceptable while
items lower in content validity were more likely to be deactivated. This
shows that on the whole, item content validity does play a role in the SMEs’
evaluation of an item's testworthiness.
The second objective of the research was to determine how SMEs and project
psychologists perceived the usefulness of the CVR forms. It became evident
that there was no universal agreement on the usefulness of the forms. Additionally,
any project psychologist biases, either for or against the use of
the forms, may have influenced how the psychologist administered the forms to
239
. .