October 2007 Volume 10 Number 4 - Educational Technology ...
October 2007 Volume 10 Number 4 - Educational Technology ...
October 2007 Volume 10 Number 4 - Educational Technology ...
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
Chang, S.-H., Lin, P.-C., & Lin, Z. C. (<strong>2007</strong>). Measures of Partial Knowledge and Unexpected Responses in Multiple-Choice<br />
Tests. <strong>Educational</strong> <strong>Technology</strong> & Society, <strong>10</strong> (4), 95-<strong>10</strong>9.<br />
Measures of Partial Knowledge and Unexpected Responses in Multiple-Choice<br />
Tests<br />
Shao-Hua Chang<br />
Department of Applied English, Southern Taiwan University, Tainan, Taiwan // shaohua@mail.stut.edu.tw<br />
Pei-Chun Lin<br />
Department of Transportation and Communication Management Science, National Cheng Kung University, Taiwan<br />
peichunl@mail.ncku.edu.tw<br />
Zih-Chuan Lin<br />
Department of Information Management, National Kaohsiung First University of Science & <strong>Technology</strong>, Taiwan<br />
u9324819@ccms.nkfust.edu.tw<br />
ABSTRACT<br />
This study investigates differences in the partial scoring performance of examinees in elimination testing and<br />
conventional dichotomous scoring of multiple-choice tests implemented on a computer-based system.<br />
Elimination testing that uses the same set of multiple-choice items rewards examinees with partial knowledge<br />
over those who are simply guessing. This study provides a computer-based test and item analysis system to<br />
reduce the difficulty of grading and item analysis following elimination tests. The Rasch model, based on item<br />
response theory for dichotomous scoring, and the partial credit model, based on graded item response for<br />
elimination testing, are the kernel of the test-diagnosis subsystem to estimate examinee ability and itemdifficulty<br />
parameters. This study draws the following conclusions: (1) examinees taking computer-based tests<br />
(CBTs) have the same performance as those taking paper-and-pencil tests (PPTs); (2) conventional scoring does<br />
not measure the same knowledge as partial scoring; (3) the partial scoring of multiple choice lowers the number<br />
of unexpected responses from examinees; and (4) the different question topics and types do not influence the<br />
performance of examinees in either PPTs or CBTs.<br />
Keywords<br />
Computer-based tests, Elimination testing, Unexpected responses, Partial knowledge, Item response theory<br />
Introduction<br />
The main missions of educators are determining learning progress and diagnosing difficulty experienced by students<br />
when studying. Testing is a conventional means of evaluating students, and testing scores can be adopted to observe<br />
learning outcomes. Multiple-choice (MC) items continue to dominate educational testing owing to their ability to<br />
effectively and simply measure constructs such as ability and achievement. Measurement experts and testing<br />
organizations prefer the MC format to others (e.g., short-answer, essay, constructed-response) for the following<br />
reasons:<br />
Content sampling is generally superior to other formats, and the application of MC formats normally leads to<br />
highly content-valid test-score interpretations.<br />
Test scores can be extremely reliable with a sufficient number of high-quality MC items.<br />
MC items can be easily pre-tested, stored, used, and reused, particularly with the advent of low-cost,<br />
computerized item-banking systems.<br />
Objective, high-speed test scoring is achievable.<br />
Diagnostic subscores are easily obtainable.<br />
Test theories (i.e., item response, generalizability, and classical) easily accommodate binary responses.<br />
Most content can be tested using this format, including many types of higher-level thinking (Haladyna &<br />
Downing, 1989).<br />
However, the conventional MC examination scheme requires examinees to evaluate each option and select one<br />
answer. Examinees are often absolutely certain that some of the options are incorrect, but still unable to identify the<br />
correct response (Bradbard, Parker, & Stone, 2004). From the viewpoint of learning, knowledge is accumulated<br />
continuously rather than on an all-or-nothing basis. The conventional scoring format of the MC examination cannot<br />
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of <strong>Educational</strong> <strong>Technology</strong> & Society (IFETS). The authors and the forum jointly retain the<br />
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies<br />
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by<br />
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior<br />
specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.<br />
95