2003 IMTA Proceedings - International Military Testing Association

of personality. Second, the same responses would be used to generate both an effectiveness score
and personality scores. Examinees who try to maximize their personality scores would risk
lowering their effectiveness scores. That is, examinees cannot ignore the effectiveness of any
actions when answering test items. Thus, an SJT that produces both an effectiveness score and
personality scale scores might be able to eliminate—or considerably reduce—faking.
Several personality traits were identified that are relevant to performing well as a Soldier
in the U.S. Army (Sager & Russell, 2003). We had developed a situational judgment test in
recent previous research for the Army (Knapp et al., 2002). The SJT score correlated
significantly not only with performance measures but also with several personality scales.
Unfortunately, attempts to develop scales based on item scores were unsuccessful. This was not
surprising considering that SJTs tend to be heterogeneous even at the item level. Thus, in the
current approach, we are developing personality scales based on the scores for individual actions.
This effort has two notable aspects. First, we generated parallel civilian situations from
military situations. Second, the test simultaneously measures both the respondent’s judgment and
several of the respondent’s personality traits. We developed descriptions of military situations
that a Soldier might experience during the first few months in the Army. For each situation, we
developed response options comprising actions that Soldiers might take in those situations. We
feared, however, that a test consisting of military situations might not be appropriate for civilian
applicants. Many applicants might not understand the situations, and those that had some
military knowledge might have an unfair advantage on the test. Therefore, we developed a
parallel civilian situation for most military situations. The remainder of this paper provides more
detailed descriptions of the development of civilian items and trait scales. The results of a recent
pilot test will also be discussed.
DEVELOPMENT OF THE MILITARY AND CIVILIAN SJTS
To develop the military SJT, we asked new Soldiers and the NCOs (Non-Commissioned
Officers) who train them to write descriptions of problem situations relevant to new Soldiers
(during the first few months in the Army). These could be actual situations they had observed or
hypothetical situations. Other Soldiers and NCOs wrote descriptions of actions that Soldiers
might take in these situations. We edited and dropped actions until there were no more than
about nine actions per situation. We asked Soldiers and NCOs to write each situation targeted to
one of the specific predictor constructs we wanted the SJT to measure.
At this point, we asked NCOs and Soldiers to write parallel civilian situations based on
the military situations. After editing these situations, we picked the best parallel civilian situation
for each military situation. Then we asked NCOs and Soldiers to write actions for the civilian
situations. We edited these actions and reduced the number of actions per situation to about nine.
We developed an initial scoring key using 10 NCOs (drill instructors and other trainers of
new Soldiers). Then we gave the draft test items (military and civilian items) to Soldiers in
training. Each item was answered by about 12 Soldiers. Based on these two data collections, some
options were dropped and a few others were edited or added. An option was dropped if the NCOs
disagreed substantially on an option’s effectiveness. Typically, options with standard deviations
above 2.00 (on a 7-point scale) were dropped. In contrast, options were also dropped if there was
541
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003