2003 IMTA Proceedings - International Military Testing Association

662
Much of the research conducted on performance measurement has been done in
the civilian performance appraisal arena, in which a supervisor evaluates a subordinate’s
performance on the job. The focus of the early research was to assess ways to improve
instruments used to evaluate performance (Arvey & Murphy, 1998; Bretz, Milkovich, &
Read, 1992; Landy & Farr, 1980). Most of this research targeted rating scales, which are
numerical or descriptive judgments of how well a task was performed. Research has been
conducted on graphic rating scales, on behaviorally anchored ratings scales (BARS), on
behavioral summary scales (BSS), and on the strengths and weaknesses of each (Cascio,
1991; Murphy & Cleveland, 1991; Borman, Hough, & Dunnette, 1976). However, the
performance appraisal research lacks studies that compare multiple rating formats. The
civilian team performance and training literature has also addressed checklist and
frequency count formats. Checklists consist of items or actions that have dichotomous
answers such as yes/no, right/wrong, or performed action versus failed to perform action,
whereas frequency counts provide an indication of the number of times that a behavior,
action, or error occurs.
However, the literature on scenario-based training lacks research on measurement
methods, on the type of data that can be collected from each, and on how measurement
purpose influences measurement method. In sum, the civilian research on rating formats
has not been conducted in the scenario-based training area, nor has it been translated into
the military arena.
Military instructors have primarily used the checklist format, which is rarely used
in the civilian sector. The reason for this difference may be that the criteria for
evaluating performance in the military may be better defined than are the criteria for
civilian jobs. That is, in the military, successful performance may be more amenable to
yes/no judgments. Furthermore, little civilian or military research has been conducted on
other rating formats, such as distance and discrepancies (D&D), which are numerical
indices of how actual performance on the task differs from optimum performance.
Moreover, after 1980, when Landy and Farr declared that further research on rating
formats was not needed, little additional research addressed this topic at all. Thus, when
to use a certain format for evaluating scenario-based training in the military and what
factors drive that decision are necessary topics of research.
To address this need, we conducted a study to provide guidance on identifying
and developing appropriate metrics for measuring human performance in military
settings. To gather information about how best to measure processes and outcomes, we
conducted brief interviews with ten experts in human performance measurement. The
literature identified a number of outcome and process measures, but we selected the ones
most relevant to scenario-based training in the military, as this domain was the focus of
the study. We used accuracy, timeliness, productivity, efficiency, safety, and effects as
our outcome measures and procedural and non-procedural taskwork and teamwork as
process measures (See Table 1 for definitions and examples.)
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003