October 2011 Volume 14 Number 4 - Educational Technology ...

from −3 to 3 for any four-option MC item, stands out for its ability to detect partial knowledge and misconceptions.
They wrote as follows: “All positive scores represent some degree of partial information and all negative scores
represent some degree of misinformation” (p. 35).
The study reported in this paper investigated the effect of CAAS using NRET as a scoring method for MC tests. The
results were based on the analyses performed on the data collected for this study. Clearly, there are a number of
limitations. First, this study used one group of students to sit for the final test under the NRET test instructions, and
the NR scores were determined from these responses. This approach of calculating different scores from one test
using one common test instruction had been employed by past researchers such as Kansup (1973) and Holmes
(2002). It has been noted that the observed score for any item of an examinee is influenced by many variables such
as guessing, testing situation, content, scoring, administration, and examinee’s behavior. Through this approach,
many of these variables can be held constant and the scoring errors can be minimized. However, further studies
could be done with two groups of students with similar ability sitting the final test, one under NRET and the other
under NR. Then, not only the reliability of NRET could be compared to NR, but the significance of a difference in
reliability could also be identified. Second, we should also compare NRET to ET. Third, mathematics was the
content matter and the study involved Form Two students only in Malaysia. Further studies conducted across
different subjects and age groups could help to clarify the generalizability of the findings of this study. Fourth, the
comparison of scoring methods should also be done using different sample distributions such as non-normal sample
distributions with different values of skewness and kurtosis. Through such studies, clearer differences between the
scoring methods may emerge. Last, but not least, this study was conducted using CAAS. Technology in assessment
often results in unforeseen social, negative and unintended consequences. Thus, further study is needed to look into
the impact on examinee’s behavior regarding these issues.
Conclusion
The results showed the feasibility of adopting NRET to replace the conventional NR. First, the analyses performed
by using the IRT models and on the performance of the low-ability students for the three most difficult items showed
that guessing was minimal using the NRET method. These findings are consistent with findings by Swineford and
Miller (1953) and Traub and Hambleton (1972), in which guessing was minimal under the penalty method.
Secondly, the analyses of NR and NRET scoring taxonomies done for the whole test, on the items with correct
answers and the items with wrong answers, showed that NRET could detect full knowledge, partial knowledge,
absence of knowledge, partial misconception, and full misconception. These results are similar with Bradbard and
Green (1986), Bradbard et al. (2004), and Chang et al. (2007), in that there is evidence that guessing is reduced and
partial knowledge can be detected. Third, the analyses performed on different subtests of the final test showed that
the NRET scores were consistently more reliable than the NR scores. This finding is consistent with suggestion by
Ma (2004), who said that when test items were scored dichotomously, potentially useful information about
individual’s level of proficiency that was contained in the other response options was lost. Thus, the precision of
measurement was reduced. The results of this study are similar to the results of the studies done by comparing ET
with NR (Collet, 1971; Hakstian & Kansup, 1975; Traub & Fisher, 1977). Although the results of NRET and ET
were comparable, NRET has an added advantage over ET. According to Jaradat and Tollefson (1988), ET
instructions are confusing despite prior practice. It is conflicting where students are taught to solve for the correct
answer but being assessed on their ability to identify incorrect answer. On the other hand, the test instructions of
NRET resemble one of the most commonly used test strategy, in which students first eliminate the obviously
incorrect options before choosing the answer.
NRET delivered through CAAS allows administrators or teachers to control the students’ responses. In this case,
there may have been several possible reasons for students to not employ the NRET test instructions. Firstly, they
may not have understood the NRET test instructions. Secondly, they may have forgotten and lapsed into the
traditional response mode. Thirdly, they may have chosen not to comply with the NRET testing mode. Regardless of
the reason for noncompliance, CAAS can ensure that students comply with the required instruction. In addition,
CAAS also allows speedy calculation of item scores and total scores. It also creates an opportunity to provide
feedback concerning performance after each item. This can help students understand the reward or penalty associated
with each response strategy. Thus, CAAS using NRET has the potential to resolve the problems of guessing and
failing to detect partial knowledge and misconception that are common with the NR method.
108