I__. - International Military Testing Association

Context Effects on Multiple-Choice Test Performance
Lawrence 5. Buck*
Planning Research Corporation, System Services
Introduction
It has long been a tenet of test construction theory and practice that test items
measuring the same content or behavioral objectives should be grou,ped within a
test. For example, Tinkelman (1971) stated:
If items measuring different content objectives or different behavioral
objectives are included in the same test, consideration should be given to
grouping the items by type. Usually the continuity of thought that such
grouping allows on the part of the examinee is found to enhance the
quality of his/her performance.
Other rationales for grouping similar items include such viewpoints as: test anxiety
may be reduced by grouping items on a test, examinees will concentrate better if
they do not jump from subject to subject, and examinees might glean information
from certain questions in a set of questions that will facilitate the answering of other
questions in the set (Gohmann & Spector, 1989).
A majority of the studies addressing item positioning have centered on the effects of
ordering questions by difficulty level rather than by content. (For a representative
sample, see: Hodson, 1984; Sax & Cromack, 1966; Leary & Dorans, 1985; and Plake,
1980.) Numerous other studies, primarily in the educational arena, have addressed
the effects of randomizing items in tests rather than presenting the items in the
order that the information is covered in the classroom or in the textbook(s). (For a
representative sample, see: Gohmann & Spector, 1989; Taub & Bell, 1975; and
Bresnock, Graves, & White, 1989).
The primary focus of this study is the effect on part and total test performance of
randomizing the items on multiple-choice tests normally constructed with the items
grouped by content areas or domains. A secondary objective was to evaluate the
effects on the individual item statistics. The items in the tests in question are
normally presented from easiest to most difficult within each domain.
Two tests were selected for this study, Rigging and Weight Testing (BM-0110) and
Outside Electrical (EM-4613). These tests are part of a testing program which
develops, administers, and maintains Journeyman Navy Enlisted Classification (JNEC)
exams for the Navy’s Intermediate Maintenance Activity (IMA) community. The tests
are part of the qualification process for special classification codes. Both the BM-
0110 and EM-4613 examinations consist of 120, four-choice, multiple-choice test
questions spread across six domains as indicated in Table I below.
Table I
Test Item-Domain Breakdown
Domains
Test # of Items 1 2 3 4 5 6
BM-0110 120 18 30 14 12 30 16
EM-461 3 120 10 6 14 55 22 13
*The author wishes to thank Norma Molina-laggard for her able assistance with the data analyses
274