January 2012 Volume 15 Number 1 - Educational Technology ...

Stability of Indices
Figure 1. Estimated marginal means of detection rates by indices
Table 2 shows a summary of one-way MANOVA results of indices for the AT condition (Wilk’s Lambda = .690, p <
.001, partial 2
= .169). As in evident from the table, except for the Wc&Bs indices, all indices performed more
sensitively on the “both spuriously high and low” aberrances than on the “spuriously high” or “spuriously low”
aberrances. Note that three IRT-based indices, lz, ECI4z, and OUTFITz, varied greatly under this condition with
partial 2
values of .21, .13, and .23, respectively. This implies that these three IRT-based indices are more unstable
across different types of aberrances than the five group-based indices did. Although ECI2z and INFITz exhibited the
lowest effect size, their detection rates were still very small. This does not make much sense. With respect to the
group-based indices, the BW indices exhibited similar detection rates to Wc&Bs for spuriously “high,” “low,” and
“both” score aberrances (Mean = .91, .89, and, .95 respectively). Since the BW indices exhibited a relatively low
effect size (partial
meaning. This indicated that the BW indices were stable across the three types of aberrances.
2
= .01), the detection rate differences among aberrant types did not have much practical
Table 2. One-way MANOVA for the types of aberrances (AT) condition (N = 81,000)
Index AT Mean SD df F partial 2
Comparisons
1
SCI
a .97 .12
2 2 1288.12*** .03 Both>High>Low
b .98 .10
3 c .92 .21
1 .96 .15
MCI 2 .96 .16 2 2993.40*** .07 Both>High>Low
3 .84 .30
1 .64 .40
NCI 2 .71 .39 2 755.71*** .02 Both>High>Low
3 .57 .42
1 .99 .05
Wc&Bs 2 .98 .11 2 258.35*** .01 High >Both= Low
3 .98 .11
1 .91 .23
BW 2 .95 .17 2 519.23*** .01 Both>High>Low
3 .89 .24
l z
1
2
3
.77
.79
.39
.35
.35
.40
2 10629.56*** .21 Both>High>Low
ECI 2 1 .00 .04 2 330.30*** .01 Both>High>Low
Z
33