January 2012 Volume 15 Number 1 - Educational Technology ...

Results
Detection rates
As seen in Table 1, although all factorial interactions were significant at the .001 alpha level, most of the interactive
2
effect sizes were small (partial ’s < .10). The maximal effect size occurred in the main effects of the index
2 (partial = .69). This implies the index factor contributed to the difference in detection rates the most. Therefore,
we examined the difference in detection rates among indices. In Figure 1, the group-based indices seemed to perform
better than the IRT-based indices. The detection rates of all group-based indices, except for the NCI, were over .90.
The detection rates of all the IRT-based indices, except for the ECI2z and the INFITz were over .50 but lower than
.70. Note that with a detection rate of .92, the BW performed as well as Wc&Bs, SCI, and MCI did.
According to the data in Table 1, the condition factors showed interactive effects on detection rates, but their effects
were very small (all were less than .02, except for the effects of the index factor on the AT and AS factors, both
partial 2
= .10). Thus, it was appropriate to investigate the stability of indices in the AT and AS conditions
individually and disregard the other interactive effects between these factors.
Table 1. Five-way analysis of variance of detection rates (N = 81,000)
Sources Type III SS df MS F Partial 2
Index (IN) 97517.79 9 10835.31 195934.65*** .69
Content Category (CC) 22.53 1 22.53 407.48*** .00
Aberrance Type (AT) 3949.63 2 1974.81 35710.52*** .08
Aberrance Severity (AS) 3859.39 2 1929.70 34894.67*** .08
Aberrance Person (AP) 418.88 2 209.44 3787.27*** .01
IN × CC 685.96 9 76.22 1378.25*** .02
IN × AT 4928.06 18 273.78 4950.78*** .10
IN × AS 4822.23 18 267.90 4844.46*** .10
IN × AP 393.93 18 21.89 395.75*** .01
CC × AT 278.80 2 139.40 2520.80*** .01
CC × AS 1002.87 2 501.43 9067.42*** .02
CC × AP 15.09 2 7.54 136.42*** .00
AT × AS 54.64 4 13.66 247.02*** .00
AT × AP 15.89 4 3.97 71.85*** .00
AS × AP 24.91 4 6.23 112.60*** .00
IN × CC × AT 550.84 18 30.60 553.38*** .01
IN × CC × AS 614.83 18 34.16 617.66*** .01
IN × CC × AP 119.45 18 6.64 120.00*** .00
IN × AT × AS 305.26 36 8.48 153.33*** .01
IN × AT × AP 43.17 36 1.20 21.68*** .00
IN × AS × AP 125.10 36 3.47 62.84*** .00
CC × AT × AS 34.02 4 8.51 153.80*** .00
CC × AT × AP 6.37 4 1.59 28.80*** .00
CC × AS × AP 23.01 4 5.75 104.02*** .00
AT × AS × AP 20.70 8 2.59 46.80*** .00
IN × CC × AT × AS 139.21 36 3.87 69.93*** .00
IN × CC × AT × AP 43.48 36 1.21 21.84*** .00
IN × CC × AS × AP 39.31 36 1.09 19.74*** .00
IN × AS × AT × AP 53.39 72 0.74 13.41*** .00
CC × AS × AT × AP 3.50 8 0.44 7.92*** .00
IN × CC × AS × AT × AP 22.32 72 0.31 5.61*** .00
Error 44763.65 809460 0.06
Total 467477.18 810000
***p < .001
32