I__. - International Military Testing Association

for MOS-specific tasks, as the percent of items answered correctly on tasks in
each area.
Hands-on job sample tests were developed to test performance on 8-14 of
the tasks selected for each MOS. The tasks that were allocated to the handson
component included, by design, both common and MOS-specific tasks, at the
target skill level as well as lower and higher skill levels, and from as many
functional areas as was feasible for testing. Scores were constructed as the
percent of steps performed correctly for a given task, averaged across the
common or MOS-specific tasks.
Examination of the supervisory tasks selected for each MOS revealed a
common structure of three areas of supervisory behaviors across the nine MOS:
Personal Counseling, Disciplinary Counseling, and Training. To measure these
three aspects of the job, simulation exercises (role-plays) were developed.
The role of a private was played by a trained civilian test scorer (three
different scorers performed the three roles for a given soldier). At the
conclusion of a role-play, the actor/scorer rated the soldier on 12-18 aspects
of behavior during the exercise. Each aspect was rated by means of 3-point
behaviorally-anchored rating scale, and an overall score was computed as the
average across the three role-plays of the mean rating on items within the
role-play.
The written situational judgment tests were designed to tap those areas
of supervisory behaviors that could not be included in the role-plays. They
were intended to evaluate the effectiveness of the NCO's judgments about what
to do in difficult supervisory situations, and were meant to tap the cognitive
aspects of first-line supervisory practice in the Army. The test contained 35
items, consisting of a situation and 3-5 alternative courses of action;
soldiers indicated which response alternatives they believed to be the most
and the least effective. Effectiveness weights were assigned to each response
of each item with the assistance of the Sergeants-Major Academy, and item
scores were computed as the weight of the soldier's "Most Effective" response
minus the weight of the soldier's "Least Effective" response. The total score
was the mean of the item scores.
Figure 1 portrays the test mode (written, job sample, and ratings) by
job component (supervisory, common task, and MOS-specific) coverage among the
testing instruments.
Test Administration and Results
Data were collected from 1009 soldiers and their supervisors (rating
scales only) in the nine MOS at 13 Army posts CONUS and in Germany. The
hands-on tests were administered by NC0 scorers under the supervision of
trained civilian staff; all other instruments were administered by trained
members of the project staff.
Table 2 gives the basic statistical characteristics for each instrument,
across the nine MOS. For every instrument, the mean scores are above the
midpoint. However, there is no great evidence of skew in the data, and the
reliability estimates are satisfactory.
543