I__. - International Military Testing Association

(1) Determine criticality of the objectives.
(2) Determine test design.
(3) Establish a minimum cutoff score.
(4) Validate the test.
Criticalitv of the obiectives
DISCUSSION
The objectives of a course are those behaviors the trainee is
expected to exhibit upon completion of training. Regardless of the
method of development, objectives are established with var.ying ._
degrees of importance or criticality. Therefore determining the
importance of the objectives must occur prior to designing the
test. While there is not an established set of procedures to
determine criticality, 'the following examples have proven to be
valid.
(1) Rank orderins of obiectives. Subject matter experts rank
the objectives from the most important to least important. This
method is most useful when courses have a few number of objectives.
(2) Yes or No. Subject matter experts determine critical-ity
by responding ltYesl' or *rN~ll. The greatest disadvantage to this
approach is that some critical objectives are more critical the
others and visa versa.
(3) Criticality based on a scale rankinu. This method uses
a set of questions to guide in determining criticality.
(a) How important is this behavior to successful
performance in the fleet?
(b) How difficult is the behavior to learn?
(c) How important is the behavior to successful
performance in the course?
A scale is normally established as O-5 or O-10. Based on the
above or similiar questions, each objective is reviewed by subject
matter experts; a number value assigned and the average calculated.
The objectives are then ranked. Objectives falling above the
established cutoff are considered critical. The cutoff score is
normally a number based on the scale used. For example, any
objective ranked 3 or above on a O-5 scale might be considered
critical. This number will vary and is based on the individual
course and its mission. This method provides the most complete way
to determine criticality. The disadvantage is that
it may be complicated and time consuming.
205