The Journal of the World Council for Gifted and Talented Children

What is measurement invariance and why does it matter?
The World Council for Gifted and Talented Children
Put in a formal definition, measurement invariance (MI) examines “the extent to which items
or subtests [or whole tests] have equal meaning across groups of examinees” (French & Finch, 2006,
p. 379). If members of one group consistently score lower than members of another group, simply
due to group membership, then the test is not yielding valid information and can be considered noninvariant.
What constitutes a “group” is not universally accepted because such definitions must be
based on the theory underpinning the topic of interest. For example, many measures of academic
achievement are highly correlated with family income (Valencia & Suzuki, 2001). Because of this, in
education, income groups should be considered for MI/E evaluation. The importance of investigating
MI/E with regard to a testing instrument cannot be overstated. An example of an instance where this
is especially important can be seen in the Naglieri Nonverbal Abilities Test (NNAT: Naglieri, 2003).
This instrument has seen widespread use by schools in an attempt to identify gifted and talented
students who come from a wide variety of backgrounds. Naglieri and Ford (2003) have made the
claim that the NNAT is culturally neutral – meaning that an individual’s membership in a traditionally
underrepresented group has no bearing on the individual’s NNAT score. Put into MI/E language, this
claim is one of NNAT invariance with regard to such factors as ELL, income, or racial / ethnic group
status. When such a claim or assumption is a critical component, as with the NNAT, the importance
of the evaluation and presentation of research regarding MI/E seems clear – this issue being noted in
the Buros Mental Measurement Yearbook review of the NNAT (French, 2005). Without such
information, practitioners and researchers have no way of knowing if the NNAT is any more or less
culturally loaded than traditional standardized ability tests of achievement or ability or specific
measure of giftedness and talent.
Typical ability or achievement tests often report how validity evidence was evaluated in the
technical manual as recommended in the Code of Fair Testing Practices in Education (JCTP, 2005)
and the Standards for Psychological and Educational Testing (AERA, 1999). However, the reporting
of MI/E or evaluation of group differences has yet to see widespread inclusion in test manuals or
reviews. The reason this causes a problem is that test users (e.g., schools, teachers, parents,
researchers) have no way of knowing if the test yields equally valid information for any one group of
students as it does for another. In the example from above, the NNAT might yield an average score
15 points lower for students from low-income families. This is not a hypothetical situation, but rather
was observed in a study by Carman and Taylor (2010). Since the NNAT is not meant to measure
income or SES, this could indicate non-invariance or bias against that particular group.
Unfortunately, mean-difference testing or general linear model methods, such as ANOVA, are not
sufficient to evaluate all aspects of a test that could suffer from non-equivalence, which is better
evaluated using latent variable models such as those in the covariance structure or structural
equation model family (Thompson & Green, 2006). Further, the origin of any observed nonequivalence
can be difficult to evaluate. In the case of the Carman and Taylor (2010) study, the
observed differences could be due either to actual group differences on the underlying construct, or
it could be the case that the instrument simply does not work in the same way for members of
different income groups (non-invariance). Without some kind of MI/E evaluation, there is no way to
know. As the NNAT stands, users and test-takers have no way to know if the instrument yields
equally valid information for students from dominant and non-dominant cultural groups alike.
Conducting measurement invariance testing using Confirmatory Factor
Analysis
Evaluating MI/ E can be approached from several different perspectives. Two of the most
common of these approach the issue from the item response theory (IRT) or structural equation
modeling (SEM) perspectives, the latter being the confirmatory factor analysis example presented
here. Those interested in the IRT approaches should consult the new book by Osterlind and Eversen
(2009) or the seminal work by Lord (1980).
The first step in conducting MI/ E testing using SEM is to specify the model to be tested. For
most cases this is simple as items on a test are written to assess certain factors or subscales. For
example, the NNAT reports a single global score, but includes four types of items: pattern
completion, reasoning by analogy, serial reasoning, and spatial visualization. Specifying which items
are meant to assess which factors serves as the model to be tested for each group separately. Even
if there is only a single factor (perhaps the general ability or “G” factor), this is still a theoretical model
100 Gifted and Talented International – 26(1), August, 2011; and 26(2), December, 2011.