I__. - International Military Testing Association

The linear model indicates that the smallest amount of variance is
accounted for by device features (.07). The combined other sources of
variance: Instructor leniency, student ability, and task difficulty,
(.21+.27+.42*.90) are predicted to mask out the variance due to the device
features. Evaluators could also artificially change their ratings to
reflect the impact of anticipated evaluation design changes. A
reexamination of summary statistics would permit evaluators to assess the
impact of hypothetical design modifications on the anticipated outcome of
the device evaluation.
DISCUSSION
Using data from a simulation model, the training effectiveness analysis
estimated that the one-trial difference between training under the visual
plus motion condition and motion alone would not be statistically
significant with a reasonable sample size (Ott, 1977). This outcome ofothe
model was confirmed through analysis of actual field data (Evans, Scott &
Pfeiffer, 1984). With this insight, from the model, the evaluator of a
.device would know in advance that control of task difficulty, student
ability, and instructor leniency in a field experiment would be necessary to
increase statistical power. True training effects attributable to the
device features are more likely to be revealed when extraneous errors are
controlled. Cochran and Cox (1957) have presented a theoretical discussion
of this problem. Instructors' rating variance, for example, may be
controlled by utilizing a standardized method for identifying when the
student has achieved mastery (Rankin & McDaniel, 1980). Some criterion
measure other than instructors' ratings could also be employed. A specific
example is automated performance measurement on the tactical range, which
unfortunately is not widely available for scientific measurement of ai,rcraft
in free flight. However, performance measurement is available in flight
simulators. Computer-aided techniques for providing operator performance
measures have been provided by Connelly, Bourne, Loental and Knoop (1974).
coNausION
This study shows that flight instructors who have knowledge of a
training situation . but who are not necessarily proficient with the
intricacies of research design and statistics can provide data useful for
planning a field experiment (device evaluation). The programs described
herein are "user-friendly" and resident in a portable microcomputer. Should
the computer be unavailable, a questionnaire could be used (Appendix). The
utility of this approach depends, in part, on asking the right questions for
a particular training environment and in part on developing the responses to
such questions into meaningful information. The model just described has
provided that utility for the present situation. Additionally, this model
may be easily adapted to other training problems involving expert ratings(see
Pf'eiffer and Horey, 1988).
197
_