I__. - International Military Testing Association

A Situational Judgment Test of Supervisory Knowledge in the U.S. Army1
Mary Ann Hanson
Personnel Decisions Research Institutes, Inc.
Walter C. Borman
The University of South Florida and Personnel Decisions Research Institutes, Inc.
A situational judgment test involves presenting respondents with realistic job situations, usually
described in writing, and asking them to respond in a multiple choice format regarding what should be
done in each situation. Situational judgment tests have been developed by other researchers to predict
job performance, especially for management and supervisory positions (e.g., Motowidlo, Carter, & Dunnette,
1989; Tenopyr, 1969).
This paper describes the development, field test, and preliminary construct validation of a situational
judgment test designed to measure supervisory skill for non-commissioned officers (NCOs) in the U.S.
Army. In contrast with most previous research the Situational Judgment Test (SJT) is a criterion measure
of job performance. It is targeted at first line supervisors (ranking E-5), and is intended to evaluate
the effectiveness of their judgments about what to do in difficult supervisory situations. Thus, the SJT is
somewhat like a job knowledge test in the supervisory part of the job. Although no research is available
on the use of situational judgment tests as criterion measures, there is research available on the usefulness
of written simulations - which are similar to situational judgment tests - as measures of professional
knowledge in fields such as law and medicine. Researchers have found that scores on written simulations
differentiate between groups with differing levels of experience or training and are often related to other
measures of professional knowledge or performance (see Smith, 1983 for a review).
Development of the SJT
Method
Development of the SJT involved asking groups of soldiers similar to the target NCOs (i.e., E-4s and
E-5s) to describe a large number of difficult but realistic situations that Army first-line supervisors face
on their jobs. Once a large number of these situations had been generated, a wide variety of possible
actions (Le., response alternatives) for each situation were gathered, and ratings of the effectiveness of
each of these actions were collected from both experts (senior NCOs) and the target group (E-5 NCOs in
beginning supervisory positions). These effectiveness ratings were used to select situations and response
alternatives to be included in the SJT. The effectiveness ratings from the senior NCOs (i.e., experts) were
also the basis for the development of SJT scoring procedures. Each of these steps is described in more
detail below.
Participants in the workshops to develop situations and response alternatives were 52 NCOs from
nine different Army posts. Some were NCOs from the target sample and some supervised target NCOs
(ranks ranging from E-5 to E-6). A variation of the critical incident technique (Flanagan, 1954) was used
to collect situations to be used as the item stems. Workshop participants were asked to write descriptions
of difficult supervisory situations that they or their peers had experienced as first-line supervisors in the
Army. This resulted in a pool of about 300 situations. Response alternatives were primarily generated by
presenting participants in later workshops with the situations that had been collected and asking them to
write, in two or three sentences, what they would do to respond effectively in that situation. This resulted
in about 15 possible responses for each situation, These responses were content analyzed and grouped to
reduce redundancies. The final result was four to ten response alternatives for each situation, with a
mean of about six response alternatives.
1 This research was funded by the U.S. Army Research Institute for the Behavioral and Social Sciences,
Contract No. MDA903-82-C-053 1. All statements expressed in this paper are those of the authors and do not
necessarily reflect the official opinions or policies of the U.S. Army Research Institute or those of the Department
of the Army.
268
._
---r
I