2003 IMTA Proceedings - International Military Testing Association

DEVELOPING APPROPRIATE METRICS FOR PROCESS AND
OUTCOME MEASURES
INTRODUCTION
Amy K. Holtzman, David P. Baker, and Robert F. Calderón
American Institutes for Research
1000 Thomas Jefferson St., NW
Washington, DC, 20007-3835, USA
aholtzman@air.org
dbaker@air.org
Kimberly Smith-Jentsch
University of Central Florida
4000 Central Florida Blvd.
P.O. Box 161390
Orlando, FL, 32816-1390, USA
kjentsch@mail.ucf.edu
Paul Radtke
NAVAIR Orlando TSD
12350 Research Parkway
Orlando, FL, 32826-3275, USA
paul.radtke@navy.mil
Scenario-based training is a systematic process of linking all aspects of scenario
design, development, implementation, and analysis (Oser, Cannon-Bowers, Salas, &
Dwyer, 1999). An exercise or scenario serves as the curriculum and provides trainees the
opportunity to learn and practice skills. In the military, scenario-based training exercises
are often used to evaluate whether individuals or teams have attained the necessary skills
for specific missions and can apply them in real-world situations. To determine if the
objectives of training have been met, performance measures can be used to assess
individual and team performance within a given scenario.
Performance measures can vary by the level of analysis, as performance can be
measured at the individual, team, and multi-team level. They can also vary by the type of
measures (i.e., outcomes or processes) and the overall purpose of the training.
Performance outcomes are the results of an individual or team’s performance, whereas
process measures “describe the steps, strategies, or procedures used to accomplish a task”
(Smith-Jentsch, Johnston, & Payne, 1998, p.62). Examples of purposes of training
include diagnosing root causes of performance problems, providing feedback, or
evaluating levels of proficiency or readiness for a task.
661
45 th Annual Conference of the International Military Testing Association
Pensacola, Florida, 3-6 November 2003