July 2006 Volume 9 Number 3 - CiteSeerX

B. Experimental Design
To evaluate the performance of the proposed STSCPSO algorithm, a series of experiments have been conducted
to compare the execution times and the solution quality of three competing approaches: STSCPSO algorithm,
Random Selection with Feasible Solution (RSFS), and exhaustive search. The RSFS program generates the test
sheet by selecting test items randomly to meet all of the constraints, while the exhaustive search program
examines every feasible combination of the test items to find the optimal solution. The platform of the
experiments is a personal computer with a Pentium IV 1.6 GHz CPU, 1 GB RAM and 80G hard disk with 5400-
RPM access speed. The programs were coded with C# Language.
To analyze the comparative performances of the competing approaches, twelve item banks with number of
candidate items ranging from 15 to 10,000 were constructed by randomly selecting test items from a computer
skill certification test bank. Table 3 shows the features of each item bank.
Table 3. Description of the experimental item banks
Item Number of test Average Average expected answer time
bank items difficulty of each test item (minutes)
1 15 0.761525 3.00000
2 20 0.765460 3.25000
3 25 0.770409 3.20000
4 30 0.758647 2.93333
5 40 0.720506 2.95000
6 250 0.741738 2.94800
7 500 0.746789 2.97600
8 1000 0.751302 2.97200
9 2000 0.746708 3.00450
10 4000 0.747959 3.00550
11 5000 0.7473007 2.99260
12 10000 0.7503020 3.00590
The experiment is conducted by applying each approach twenty times on each item bank with the objective
values (Zk) and the average execution time recorded. The lower bounds and upper bounds of testing times are 60
and 120 minutes, respectively, and the maximal number of common test items between each pair of test sheets is
5. To make the solutions hard to obtain, we set the target difficulty level D = 0.5 which sufficiently deviates
from the average difficulty of item banks. The STSCPSO algorithm is executed with 10 particles for 100
generations. The execution time of RSFS is set the same as that of the STSCPSO algorithm, while the maximal
execution time of the exhaustive search is set to 7 days to obtain the optimal solution.
STSCPSO
Table 4 Experimental results
RSFS Optimum Solution
N Zk Average
Time (sec)
Zk Average
Time (sec)
Zk Average
Time (day)
15 0.44 50 0.62 50 0.42 2 days
20 0.44 63 0.70 63 - > 7 days
25 0.44 80 0.66 80 - -
30 0.44 102 0.68 102 - -
40 0.40 134 0.76 134 - -
250 0.36 815 0.54 815 - -
500 0.28 1805 0.68 1805 - -
1000 0.22 3120 0.46 3120 - -
2000 0.18 6403 0.60 6403 - -
4000 0.16 12770 0.62 12770 - -
5000 0.16 15330 0.54 15330 - -
10000 0.14 21210 0.56 21210 - -
11