I__. - International Military Testing Association

EVALUATING TRAINING PROGRAM MODIFICATIONS
Deborah Lawson McCormick and Paul L. Jones
Naval Technical Training Command
Evaluating changes in training programs is never a simple
task, even under laboratory conditions where threats to validity
can be controlled. In operational settings evaluation may appear
to be an insurmountable problem -- one in which good evaluation
methodology does not seem feasible. One major problem for __
evaluators in operational settings is that they are often not
consulted until after training modifications have already been
initiated. As a result, both experimental control and
opportunities for data collection are severely limited.
Even in those rare cases where evaluators are a part of the
implementation from its onset, problems exist. For example, an
evaluation design which uses equivalent control and experimental
groups is often not possible in on-going training programs. In
addition, operational settings are inherently dynamic environments;
consequently, the effects of deliberate program changes are
confounded with effects of other random factors which 'constantly
impact the program. In these cases, isolating effects directly and
unquestionably attributable to factors of the program change is
impossible.
This difficulty in establishing definite cause and effect
relationships is sometimes used as a reason to forego evaluation.
Rather than attempting a seemingly futile task, the tendency is to
rely on intuition. The argument goes something like this: "These
changes make sense, the students like them, the instructors like
them . . . they probably work."
However, increased competition for funding dollars makes the
need to verify training improvement and justify additional funds
crucial. Increasingly, funding sources are requiring hard data in
support of dollars spent. As evaluators, we are being forced to
accept that a less than perfect evaluation (that is, one which only
suggests, rather than "proves," cause and effect) is better than no
evaluation at all.
This paper describes an evaluation model which we feel is
flexible enough to prove useful in most evaluation circumstances,
from the ideal condition, where evaluation has been planned in
conjunction with change implementation, to those evaluation
nightmares, where change implementation is complete before the
evaluator is consulted. Following a brief description of the
model, and application of its use is discussed.
122