I__. - International Military Testing Association

1
Interrater Reliability as an Indicator of
HOPT Quality Control EffeCtiVeAeSs
Major P. J Exner, USMC
HQ USMC
Jennifer L. Crafts
Daniel B. Felker
Edmund C. Bowler
American Institutes for Research
Paul W. Mayberry
Center for Naval Analyses
The United States Marine Corps Job Performance Measurement
Project is attempting to validate enlistment quality requirements
against actual on-the-job requirements. Since there are nearly
500 Military Occupation Specialties (MOSS), developing hands-on
performance tests (HOPTS) for each MOS is impractical. Therefore
the Marine Corps has elected to test relatively large numbers of
Marines in a few critical MOSS in each of the four Armed Services
Vocational Aptitude Battery composites used for classification.
Testing began with the General Technical (GT) composite in
1986-87 for the infantry occupational field. In 1989-90 tests for
Mechanical Maintenance (MM) composite MOSS were developed and
administered. In August, 1990, hands-on testing was completed on
approximately 1900 Marine automotive and helicopter mechanics.
Because of the many possible sources of error in the
development and administration of HOPTS, quality control is
critical at every step. Poor test design or execution can
significantly reduce validities and diminish the value of the
results. In a preliminary Marine Corps study Maier (1988)
reported a large reduction in validities due to various errors.
Such errors can include content, test design, test administrator
(TA) training, environmental, temporal, and other effects. One
indicator of possible problems is interrater reliability,.. or TA
agreement.
In this paper, we will'review the quality control
measures used in MM testing and examine preliminary reliability
results across task, test site, MOS, and time.
A series of quality control measures were used to ensure the
quality of hands-on performance data. They include: recruitment
of former or retired Marines to serve as TAs; selection of TA
applicants based on scores on structured interviews; standardized
test site setup;- extensive and ongoing training of TAs; rotation
This research was funded by Contract No. N00014-87-C-0001 and by
subcontract CNA 4-89. ~11 statements expressed in this paper are
those of the authors and do not necessarily reflect the official
views or policies of the Department of the Navy or the U.S. Marine
Corps.
535