October 2007 Volume 10 Number 4 - Educational Technology ...

Testbank management system
The testbank management system is normally accessed by instructors who are designing test items, revising test
items, designing examinations, and reusing tests. The main advantage of the item banking is in test development.
This system allows a curriculum designer to edit multiple-choice questions, constructed-respond items, and true/false
items and specify scoring modes. Designed test items are stored in the testbank database. Test items can be displayed
in text format or integrated with multimedia images. This system supports three parts of the preparation of
examinations: (1) creating original questions, (2) browsing questions from a testbank, and (3) using questions from a
testbank by selecting questions items at random. The system provides dichotomous scoring and partial scoring for
MC items. After the item sheet has been prepared, the curriculum designer must specify “examinee eligibility,”
“testing date,” “scoring mode,” “time allowed,” and “question values.” The computer-based assessment system
permits only eligible examinees to take the test at the specified time. Although the system can automatically and
instantly grade multiple-choice and true/false questions, the announcement of testing scores is delayed until all
examinees have finished the test.
Test diagnosis system
The test diagnosis system analyzes items and examinees’ ability based on scoring methods (dichotomous scoring and
partial scoring) and the data retrieved from the answer record and gradebook database. The diagnostic process is
summarized as follows: (1) Sorting the results from the test records. The system first sorts the item-response patterns
for all examinees. The answer records are rearranged according to each examinee’s ability and the number of
students who have given the correct answer to a given question. (2) Matrix reduction. The system then deletes
useless item responses and examinee responses, such as those questions to which all examinees gave correct answers
or wrong answers, or received a full or a zero score, since these data do not help to assess the ability of examinees
(Baker, 1992; Bond & Fox, 2001). Matrix reduction can reduce the resources and time required to conduct the
calculation. (3) The JMLE procedures (Hambleton & Swaminathan, 1985; Wongwiwatthananukit, Popovich, and
Bennett, 2000) depicted in Figure 2 are applied to estimate the ability and item parameters. Then the diagnosis
procedures are followed by the item and person fit analysis, and the procedures are described as follows: First, let
random variable X ni denote examinee n’s response on item i, in which X ni = 1 signifies the correct answer; θ n is
the personal parameter of examinee n; and b i denotes the item parameter, which determines the item location and is
called the item difficulty in attainment tests. The expected value and variance of X ni are shown in equations (1) and
(2).
(1) EX ( ni) = exp( θn − bi) /[1+ exp( θn − bi)]
= πni
(2) Var( X ) = π (1 − π ) = W
ni ni ni ni
The standardized residual and kurtosis of X ni are shown in equations (3) and (4).
(3) Zni = [ Xni − E( Xni )]/[ Var( Xni
)]
(4) C
4 4
= [(1 − π ) ( π )] + [(0 −π ) (1 − π )]
ni ni ni ni ni
The person fit analysis shows the mean square (MNSQ) and standardized weighted mean square (Zstd), as
represented in equations (5) and (6).
1 2
2
(5) MNSQ = ∑WniZni ∑ Wni = ν n
i i
13
(6) Zstd = tn = ( ν n − 1(3/ ) qn) + ( qn
/3) where = ∑( − ) ∑
q C W ( W )
2 2 2
n ni ni ni
i i
The item fit analysis shows the mean square (MNSQ) and standardized weighted mean square (Zstd) are as follows:
101