2003 IMTA Proceedings - International Military Testing Association

542
too much agreement among the Soldiers in training. Finally, we narrowed down the options to
about seven per situation. Where possible, a set of options were selected for a situation so that there
was a wide range of keyed effectiveness values (computed as the mean of the NCO ratings).
DEVELOPMENT OF THE TRAIT SCORING
The theoretical basis for using the SJT to measure both traits and judgment is based upon
the following model. When an examinee judges the effectiveness of an action, that judgment is
determined by both the examinee’s personality and his/her knowledge/training/experience relevant
to the situation. The traditional SJT score taps the examinee’s knowledge/training/experience
whereas the trait scores tap part of the examinee’s personality.
As mentioned above, SJT tests are heterogeneous. Therefore, we decided to measure
traits at the lowest level possible: the individual option. Nineteen individuals with graduate
degrees in industrial-organizational psychology were recruited to rate the traitedness of each
response option. Each response option was rated by five to seven psychologists. For each traitoption
combination, participants rated the degree to which the action and trait were related.
Inverse relationships were given negative ratings. Each point in the rating scale represented a
range of correlations. The mean (across psychologists) rating represented the traitedness for that
trait on that option.
PILOT TEST RESULTS: JUDGMENT SCORES
Eight draft test forms were given to 319 Soldiers in U.S. Army reception battalions.
These Soldiers had just entered the Army but had not been assigned to training yet. Therefore,
they would be similar to applicants. Each Soldier completed one civilian SJT form (A–D) and
one military SJT form (1–4). There were four pairings of forms: A-1, B-2, C-3, and D-4. Within
each form-pair, the order was randomized. That is, half of the Soldiers got the military form first;
the other half got the civilian form first. Most items had seven response options. The civilian
forms had 14–16 items; the military forms had 11–13 items. There was no attempt to put a
military item and its parallel civilian item within the same form-pair.
The Soldiers responded by rating the effectiveness of each option on a 7-point scale
(where higher numbers represent greater effectiveness). The judgment score for an option was
computed as shown in Equation 1 below. The difference between the rating and keyed
effectiveness values is subtracted from 6 so that higher values represent better scores. The
judgment score for an entire test form was merely the mean of the option scores.
optionEffectivenessScore = 6 – | SoldiersRating – keyedEffectiveness | (1)
The reliability of the judgment scores was estimated via coefficient alpha. Table 1 shows
these values for each of the eight forms. The reliability estimates are around .90. Table 1 also
shows that the judgment score is measuring essentially the same thing on the civilian and
military forms. The correlations between forms are almost as high as the reliability estimates.
The correlation rc estimates the correlation between the constructs measured in the two forms.
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003