I__. - International Military Testing Association

TEST DESIGN AND MINIMUM CUTOFF SCORES
Sandra Ann Rudolph, Training Appraisal
Chief of Naval Technical Training
INTRODUCTION
It has become increasing obvious in the last few years that
the United States government cannot continue to operate with little
concern for who will pay the bill. The apparent message is to
do better with less. This means we must become more efficient in
our way of conducting business. For many of us--our business,
is training. Being efficient means we must use our resources
wisely for the purpose intended. In training our resources are
numerous--training devices, curriculum, instructors --while our
purpose is solitary--provide the training necessary for graduates
to perform in the fleet. While performance is the key, there is
background knowledge that is necessary for the trainee to grasp the
performance.
BACKGROUND
In the training environment of yesterday, where money was no
object, training was easier. There was little concern for statistical
evaluation, effectiveness, or efficiency. We trained by
the seat of our pants--experience wasn't the best teacher, it was
the ONLY teacher. Today, lack of attention in these areas could
mean loss of training dollars. One of the big areas of concern
deals with attrition--or the dropping of trainees from,a designated
training program. While there are many causes for attrition,
recent attrition analysis visits to such schools as Air Traffic
Control School, Music School, and Boiler Technician /Machinist Mate
School, indicate that testing programs may be at the very core of
many of our problems. The following questions were used to
determine how knowledge testing was being used to measure success:
(1) Have critical course objectives been identified with
corresponding emphasis on testing?
(2) Have the knowledge tests been designed to measure the
objectives to the learning level required?
(3) How was the minimum cutoff score for the knowledge
tests determined?
(4) Has the test design and cutoff score been validated?
(5) Have alternate versions of the tests been developed
that are consistent with the valid test design?
It became apparent that testing was a problem. It was
discovered that the emphasis and training had been placed on
individual test-item development and test-item analysis, not on
test development and test analysis. In other words, there was no
assurance that the objectives were being tested nor any evidence
on how the cutoff score was determined. To standardize the
approach to test design, the following process was established:
204