I__. - International Military Testing Association

-
tion procedure to assess a job knowledge test being used as a criterion measurement
of psychiatric aide training success. They stated, "It is evident
that the content validity could be improved in subsequent revisions if the
method is used as part of the test construction process.”
Kesselman and Lopez (1979) developed an accounting job knowledge test using
the procedure which they found to be superior to a commercially available
mental ability instrument in predicting two criteria; supervisor assessment
of subordinate job knowledge and supervisor assessment of overall job perfor-
mance.
Distefano, Pryer, and Erffmeyer (1983) showed that a variation of the Lawshe
method could be used in the development of a behavioral rating scale of job
performance, while providing quantitative content validity evidence of, the
criterion scale.
Finally, Carrier, Dalessio, and Brown (1990) used Lawshe’s three-point scale
to focus on the correspondence between inferences made using content validity
and criterion-related validity strategies. They found that for experienced
candidates, job experts seemed to be able to identify those items on an interview
guide that predicted the commissions of personnel into the Life Insurance
Marketing and Research Association. They also noted that ". . .using
content validity as the sole evidence for test validity should be limited to
situations where test developers are working with well-defined constructs,
such as acquired skills or specific knowledge.”
Method
Two content validity rating (CVR) forms were developed using a Lawshe-type
scale, one form fo.r development of the SKT taken for promotion to E-5 and one
for the development of the SKT taken for promotion to E-6 and E-7. The
USAFOMS forms incorporated minor adjustments to the Lawshe approach. In par-
titular, it was necessary to reference the grade level of the test, since
knowledges required for successful performance of the E-5 duties may be considerably
different from those required to perform E-6 and E-7 duties. In
addition, the USAFOMS forms focused on successful performance in the specialty,
not just in the job, a much broader view, since a specialty encompasses a
family of related jobs.
Whereas Lawshe used a test of statistical significance with the panel's ratings,
this was not practical at USAFOMS because of the small number of individual
raters being used. Only two to six SMEs normally participate in a
test development project. To require statistical significance with such a
small sample would require unanimous agreement of item essentiality. Rather
than impose strict statistical criteria with the new ratings, the USAFOMS
policy was stated as follows: YVRs will be used to encourage SMES to focus
first on the appropriateness of test item content as it relates to successful
performance in the specialty." SMEs were not required to take special actions
as a result of the various ratings. In essence, SMEs could retain (ei-
ther reuse on the next revision of the test or designate as an alternate) or
deactivate (designate as unacceptable for reuse) an item without regard to
their ratings on the CVR forms. There are, however, other requirements such
as clear reference support and acceptable item statistics that must be met if
an item is to be retained.
This research examines how Lawshe's procedure, with the noted modifications)
was employed at USAFOMS -- an organization whose promotion tests impact most
236
.