I__. - International Military Testing Association

MILITARY TWTING ASSOCIATION
iYH) Annuiil timkrcnw
FORECASTING TRAINING EFFECTIVENESS (FORTE)
Mark G. Pfeiffer and Richard M. Evans
.--- .-- - -- -
Naval Training Systems Cent.er and Training Ferformance Data C:ent.er
Orlando, FL
A model was developed to simulate a variety of aviation training device
evaluation outcomes. This simulation model is designed to explore sources
of error threatening the sensitivity of device evaluations. Selection of ._
evaluation designs is guided by a model that elicits information from
experienced flight instructors. This practical knowledge is transformed
into data that are used i,n simulating a training effectiveness evaluation.
Effects of variables such as instructor leniency, task difficulty, and
student ability are estimated by two different methods. Available in the
output is an estimate of transfer ratios based on trials-to-mastery, a
diagnosis of deficiencies, an exploration of possible sources of variance,
and an estimate of statistical power and required sample size. Finally, all
data analyses can be accomplished in less than 2 man-days and prior to the
actual field experiment. Estimates of accuracy, reliability, and validity
of the model are high and in an acceptable range.
Backwound
Major sources of error variance that can mask the true contribution of
a training device to training effectiveness include instructor leniency,
student ability, and task difficulty (McDaniel, Scott & Browning, 1983).
First, instructors' grades are often unreliable criterion measures. Next,
individual abilities among students vary widely. Finally, tasks vary
greatly in difficulty level. Some tasks can be mastered by students in one
or two trials, while others may require 30 trials. These sources of
variance make ratings of students' performance insensitive measures of
training device effectiveness. However, the magnitude can be identified
with sensitivity analysis prior to actual field experiments.
Sensitivity Analysis
Sensitivity analysis is a planning technique (Lipsey, 1983) which
focuses on the impact of variance on variables of interest. The device
evaluation must be carefully planned if the results are to have practical
value and show a true difference between experimental and control groups.
During the planning phase for device evaluations an investment in time may
help identify the problems that introduce unwanted error variance into the
device evaluation. Performance data qenerated by flight instructors can be
used for this purpose.
The basic framework of the present "sensitivity" analysis differs from
that described by Lipsey (1983) in that it employs the "insensitive"
instructor's rating of students as a performance measure. Lipsey .wouid
rather seek a more sensitive measure. While this rating measure may not be
a particularly good psychometric measure, it is dictated by operational
constraints. Instructors' ratings are used extensively in the transfer of
training literature.
“Approved for public release; distribution is
unlimited.”
191