October 2007 Volume 10 Number 4 - Educational Technology ...

Research hypotheses
This study proposes four hypotheses based on literature reviews. First, Alexander et al. (2001) explored students in a
computer technology course who completed either a PPT or a CBT in a proctored computer lab. The test scores were
similar, but students in the computer-based group, particularly freshmen, completed the test in the least amount of
time. Bodman and Robinson (2004) investigated the effect of several different modes of test administration on scores
and completion times, and the results of the study indicate that undergraduates completed the computer-based tests
faster than the paper-based tests with no difference in scores. Stated formally:
Hypothesis 1. The average score of CBTs is equivalent to that of PPTs.
Second, the statistical analysis of Bradbard et al. (2004) indicates that the Coombs procedure is a viable alternative to
the standard scoring procedure. In Ben-Simon et al.’s (1997) classification, NS performed very poorly in
discriminating between full knowledge and absence of knowledge. Stated formally:
Hypothesis 2. ET detects partial knowledge of examinees more effectively than NS.
Third, Bradbard and Green (1986) indicated that elimination testing lowers the amount of guesswork, and the
influence increases throughout the grading period. Stated formally:
Hypothesis 3. ET lowers the number of unexpected responses for examinees more effectively than NS.
Finally, most items in mathematics and chemistry testing need manual calculation. The need to write down and
calculate answers on draft paper might lower the answering speed (Ager, 1993). Stated formally:
Hypothesis 4. Different types of question content, such as calculation and concept, influence the performance of
examinees on PPTs or CBTs.
Research variables
The independent variables used in this study and the operational definitions are as follows:
Scoring mode: including partial scoring and conventional dichotomous scoring to analyze the influence of
different answering and scoring schemes on partial knowledge.
Testing tool: comparing conventional PPTs with CBTs and recognizing appropriate question types for CBTs.
The dependent variable adopted in this study is the students’ performance, which is determined by test scores.
Experimental design
Tests were provided according to the two scoring modes and two testing tools, which were combined to form four
treatments. Table 2 lists the multifactor design. Treatment 1 (T1) was CBTs, using the ET scoring method;
Treatment 2 (T2) was CBTs, using NS scoring method; Treatment 3 (T3) was PPTs, using ET scoring method; and
Treatment 4 (T4) was PPTs, using NS scoring method.
Data collection
Table 2. Multifactor design
CBTs PPTs
ET T1 T3
NS T2 T4
The subjects of the experiment were 102 students in an introductory operations management module, which is a
required course for students of the two junior classes in the Department of Information Management at National
Kaohsiung First University of Science and Technology in Taiwan. All students were required to take all four exams
103