January 2012 Volume 15 Number 1 - Educational Technology ...

Huang, T.-W. (2012). Aberrance Detection Powers of the BW and Person-Fit Indices. Educational Technology & Society, 15 (1),
28–37.
Aberrance Detection Powers of the BW and Person-Fit Indices
Tsai-Wei Huang
Department of Counseling, National Chiayi University, 85 Wunlong Village, Minsyong, Chaiyi 62103, Taiwan //
twhuang@mail.ncyu.edu.tw
ABSTRACT
The study compared the aberrance detection powers of the BW person-fit indices with other group-based indices
(SCI, MCI, NCI, and Wc&Bs) and item response theory based (IRT-based) indices (OUTFITz, INFITz, ECI2z,
ECI4z, and lz). Four kinds of comparative conditions, including content category (CC), types of aberrance (AT),
severity of aberrance (AS), and the ratios of aberrant persons (AP), were implemented under the tolerance of a
.05 false positive rate. Results showed that group-based indices performed better than IRT-based indices.
Although the BW indices and most of the other group-based indices exhibited over 90% detection rates, the BW
indices exhibited the best stability across implemented conditions. On the basis of their highly stable detection
power and objective cutoffs, the BW person-fit indices were recommended for use in diagnosing students’
learning issues in classrooms.
Keywords
Person-fit index, BW indices, group-based indices, IRT-based indices, detection power
Introduction
The indices that have been developed to detect aberrant response patterns are referred to as unusual response
indicators, caution indices, fit indices, aberrance indices, appropriateness measurement indices, or likelihood indices
(Meijer & Sijtsma, 1995; D’Costa, 1993a, 1993b). According to measurement theory, some indices are group based,
and some are based on item response theory (IRT; Harnisch & Linn, 1981; Kogut, 1986, Meijer & Sijtsma, 1999).
Group-based indices refer to those indices that use certain group characteristics (e.g., the concept of item difficulty or
the proportion of correct item responses to the total number of responses) to identify aberrances. On the other hand,
most IRT-based indices measure the degree of consistency of an observed response pattern with respect to a certain
IRT model used.
Most group-based aberrance indices, however, encounter the problem of without knowing their theoretical
distributions such that some alternative approaches, like rules of thumb, are provided to enable clinical use. For
example, the original Sato caution index (SCI; Sato, 1975) was deemed as aberrant when it was higher than 0.5.
Harnisch and Linn (1981) later proposed the modified caution index (MCI) values and suggested that values greater
than 0.3 should be considered aberrant. For the within-ability-concern and beyond-ability-surprise indices (Wc&Bs)
which were introduced by D’Costa (1993a, 1993b), values between 0.3 and 0.5 necessitated “routine caution,” and
values greater than 0.5 required “serious caution.” Consequently, the lack of absolute cut-off standards results in the
sample-dependent identification and interpretation of aberrant responses, and the term of “index” may even be
questioned for these indices. On the other hand, although some IRT-based indices have been standardized in order to
examine their null-hypothesis-based distributions, the challenge of approximating corresponding distributions—
usually normal distributions—still exists under the assumption of large samples. In other words, the use of these
indices is only appropriate for large samples and cannot be guaranteed to be appropriate for small samples.
Therefore, it is questionable to use these asymptotic-distribution-based indices to infer aberrance when sample sizes
are small, especially when the data sets do not have normal distributions.
Two group-based indices, the beyond-ability-surprise index (B) and the within-ability-concern index (W) both
inheriting from the Wc&Bs indices, can apply cut-off standards in small samples (Huang, 2007). The B index was
designed to detect the “beyond-ability” aberrant response patterns and the W index detected those that are “withinability.”
The beyond-ability response pattern is assessed using a Guttman scale: it measures the surprise of a person
when he/she correctly answers items beyond his/her ability level. Someone exhibiting a within-ability response
pattern is considered to need more attention because some of their wrong answers are below their ability levels. The
aberrances cutoffs provided by the BW indices are based on a permutation technique that is norm-referred for each
ability-ratio/error-ratio (or T/K-E/K) cell (Huang, 2007). They are established according to various ability ratios and
error ratios under three types of percentiles (90%, 95%, and 99%). Any observed B or W values greater than the
ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the
copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies
are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by
others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from the editors at kinshuk@ieee.org.
28