I__. - International Military Testing Association

EVALUATION MODEL
The evaluation system we recommend approaches evaluation on
two levels. At the more immediate level, we attempt to determine
effects on student performance in the specific training areas
modified. For example, changes in test scores or training time in
those specific content areas might be analyzed. The second level
attempts to determine how the training program as a whole was
affected by the modifications. Such measures as course attrition,
total training time, performance in other areas of the program,
etc., would be considered. Inferring cause and effect
relationships becomes riskier as one moves to these more general
measures of effect -- measures further removed from the proposed -'
cause. However, modification in one area of the training program
should ultimately affect the program as a whole and become
manifested in these general measures. In reality, it is this
broader impact which serves as the bottom line point of interest
for most of our clients.
The evaluation model we use can be condensed to a six-step
process described below:
1. Beqin evaluation nlanninq earlv, before imnlementation of the
proqram chanqe if oossible. The evaluator should be involved as
soon as possible, ideally during the modification planning stage -certainly
prior to modification implementation. Many threats to
validity can be anticipated and controlled if the evaluator is
involved in this manner. Realistically, however, we know that this
scenario seldom occurs. More often, the evaluator is called in
after the modification has been implemented. For this reason, we
usually find ourselves beginning with step two.
2. Know the oroqram YOU are about to evaluate. A thorough
understanding of the nature of the program change and its impact on
the general operation of the training program is critical to good
evaluation. The evaluator must understand the program's
objectives, the anticipated impact of the change on these
objectives, and the methods used to accomplish them. In addition,
the evaluator must determine what data is currently being collected
to evaluate program performance and whether this data might be
useful in evaluating the program change. Most importantly, a
definitive statement of how the change is intended to affect the
program (that is, the goal of the program chanqe) must be
formulated.
3. Determine data collection procedures and qather baseline data.
The purpose of baseline data is to develop a snapshot of how well
the program is performing in the area to be modified prior to the
change. Often you will find existing measures of performance, such
as test scores, which directly address this question. In other
123
I