October 2011 Volume 14 Number 4 - Educational Technology ...

answer. However, students can choose either “correct,” or “wrong,” or “not sure” for any option. The scoring for
NRET is based on the combination of NR and ET scoring methods. For any MC item with four options, one point is
awarded for each wrong option eliminated correctly. However, a penalty of three points is deducted if the correct
answer is eliminated. One additional point is awarded if the answer chosen is correct, and no point is giving for
choosing “not sure.” Both the penalty and the choice of “not sure” are included to discourage guessing. Thus, the
NRET score for any MC item with four options ranges from −3 to 4. Table 1 below contains the test instruction and
scoring guides for NRET.
Table 1. NRET test instruction and scoring guides
Test Instruction
MUST choose ONE option as the ANSWER by using “ CORRECT.”
ELIMINATE option(s) that you are SURE ARE NOT THE ANSWER by using “X
WRONG.”
USE “? NOT SURE” if you are NOT SURE of an option.
You have the flexibility to choose NONE (0), ONE (1), TWO (2), or THREE (3) “X
WRONG,” or “? NOT SURE.”
Scoring Guides
ONE (1) point awarded if the option with “ CORRECT” is the correct answer.
ONE (1) point awarded for each option eliminated correctly with “X WRONG.”
A PENALTY of 3 points deducted if the correct answer is eliminated with “X WRONG.”
Your score will range from –3 to 4.
Increasingly, computer-based testing (CBT) is also being used in the classroom as computers and Internet
accessibility become pervasive. Thus, studies related to CBT become important. However, most of these studies are
still mainly based on NR or alternative scoring methods with complex and unfamiliar test instructions (He &
Tymms, 2005).
Purpose of the study
Therefore, in this study, a computer-adaptive assessment software (CAAS) using the NRET scoring method was
developed and tested in Malaysian secondary schools in 2008. The main aim was to investigate the robustness of
NRET scoring method in reducing guessing and detecting partial knowledge and misconceptions. In particular, this
study investigated the following four research questions:
1. Research question 1: What was the extent of guessing under NRET method?
2. Research question 2: Could NRET method detect partial knowledge and misconceptions?
3. Research question 3: What was the reliability of NRET scores as compared to NR scores?
Review of related literature
This section briefly reviews the literature related to MC tests, different scoring methods, and CBT. Major
weaknesses of NR scoring are highlighted. In addition, this section also discusses various measures taken by the
different scoring methods and the use of technology to improve these weaknesses.
Multiple-choice and scoring methods
Oral examination was the primary means of educational testing before the mid-nineteenth century. Written tests in
the form of essay questions were introduced to replace oral examinations. Studies done in the early part of the
twentieth century showed that essay tests tended to be highly subjective and unreliable in measuring students’
performance. These findings motivated educators to develop more objective educational measurements. MC was first
used in 1917 for the selection and classification of military personnel for the United States Army (Ebel, 1979).
Today, MC tests are the most highly regarded and widely used type of objective tests for measuring knowledge,
ability, or performance (Ben-Simon et al., 1997). Traditionally, NR is the scoring method in which the number of
100