I__. - International Military Testing Association

One-hundred and eighty of the most promising situations were then chosen based on their content
(e.g., appropriately difficult, realistic, etc.) and the number of plausible response alternatives available.
For each of these 180 situations retained, information concerning the effectiveness of the various response
alternatives was collected from two groups, a group of expert NCOs and a group of the target
population NC0 job incumbents. The expert NCOs were 90 students and instructors at the United States
Army Sergeants Major Academy. These NCOs were among the highest ranking enlisted soldiers in the
Any (rank of E-8 to E-9), and all had extensive experience as supervisors in the Army. The target
NCOs were 344 second tour soldiers (rank of E-4 to E-5) who were participating in a field test of a group
of job performance measures at several Army posts in the United States and Europe. For each SJT situation,
these respondents were asked to rate the effectiveness of each response alternative on a seven point
scale (1 = least and 7 = most effective). Because there were still 180 situations and time limitations, each
soldier could only respond to a subset of the situations. This resulted in about 25 expert NC0 and 45
incumbent NC0 responses per situation.
Items (situations) for the field test version of the SIT and response alternatives for these items were
then selected based on these data, The following criteria were used to select 35 of these situations and
from 3-5 response alternatives for each situation: 1) the expert group had high agreement concerning the
most effective response for the item; 2) the item was difficult for the incumbents (i.e., agreement was
substantially lower than for the expert group); 3) the difference between the expert and the incumbent
responses for each situation was judged to reflect an important aspect of supervisory knowledge; and 4)
the content of the final group of situations was as representative as possible of the first-line supervisory
job in the Army.
Field Test of the SJT
The field test of the SJT had three major objectives. The first objective was to explore different
methods of scoring the SJT. The second objective was to examine and evaluate the psychometric properties
of this instrument. The final objective was to obtain preliminary information concerning the consttuct
validity of the SJT as a criterion measure of supervisory job knowledge.
The SJT was administered as part of a larger data collection effort to a sample of 1049 NCOs (most
were E-4s and E-5s) at a variety of posts in the United States and Europe. For each of the 35 SJT items,
these soldiers were asked to place an “M” next to the response alternative they thought was the most
effective and an “L” next to the response alternative they thought was the least effective.
Scoring Procedures. Several different procedures for scoring the SJT were explored. The most
straightforward was a simple number correct score. For each item, the response alternative that had been
given the highest mean effectiveness rating by the experts (senior NCOs) was designated the “correct”
answer. Respondents were then scored based on the number of items for which they indicated that this
“correct” response alternative was the most effective. The second scoring procedure involved weighting
each response alternative chosen by soldiers as the most effective by the mean effectiveness rating given
to that response alternative by the expert group. This gives respondents more credit for choosing
“wrong” answers that are relatively effective than for choosing wrong answers that are very ineffective.
These item level effectiveness scores were then averaged to obtain an overall effectiveness score for each
soldier, Averaging these item level scores instead of simply summing them placed respondents’ scores
on the same 1 to 7 effectiveness scale as the experts’ ratings and ensured that respondents were not penalized
for any missing data (up to 10% missing responses were allowed).
Scoring procedures based on respondents’ choices for the least effective response to each situation
were also explored. The ability to identify the least effective response alternatives might be seen as an
indication of respondents’ ability to avoid these very ineffective responses or in effect to avoid “screwing
up”, As with the choices for the most effective response, a simple number correct score was computed:
the number of times each respondent correctly identified the response alternative that the experts rated
the least effective. In order to differentiate this score from the number correct score based on choices for
the most effective response, this score will be referred to as the L-Correct score, and the score based on
choices for the most effective response (described previously) wit1 be referred to as the M-Correct score.
Another score was computed by weighting respondents’ choices for the least effective response altema-
269