ICCS 2009 Technical Report - IEA

Chapter 11:Scaling procedures for ICCS test itemsWolfram Schulz and Julian FraillonIntroductionThis chapter describes the procedures used to analyze and scale the ICCS international andregional test items that were administered to measure students’ civic knowledge. The chaptercovers these topics:• The scaling model used to analyze and scale the test items;• Test coverage and item dimensionality;• Assessment of item fit;• Assessment of scorer reliabilities for open-ended items;• Differential item functioning by gender;• Review of cross-national measurement equivalence;• International item adjudication;• International item calibration and test reliability;• International ability estimates (plausible values and weighted likelihood estimates);• Estimation of changes in civic content knowledge between 1999 and 2009; and• Regional test items for the European and Latin American modules.The development of the ICCS test items was described in Chapter 2 and was guided by theICCS assessment framework (see Schulz, Fraillon, Ainley, Losito, & Kerr, 2008).The scaling modelItem response theory (IRT) scaling methodology was used to scale the test items.Use of the one-parameter (Rasch) model (Rasch, 1960) for dichotomous items means that theprobability of selecting Category 1 instead of 0 is modeled asP i (q) = exp(q n– d i )1+exp(q n – d i ) ,where P i (q) is the probability for person n to score 1 on item i, q n is the estimated ability ofperson n, and d i is the estimated location of item i on this dimension. For each item, itemresponses are modeled as a function of the latent trait q n .In the case of items with more than two (k) categories (as, for example, with Likert-type items),this model can be generalized to the partial credit model (Masters & Wright, 1997), whichtakes the form ofexpS(q n – d i + t ij )k=0P xi (q) = m i kx i =0,1,…,m i .S expS(q n – d i + t ij )h=0xk=0Here, Px i (q) denotes the probability of person n scoring x on item i, and q n denotes theperson’s ability. The item parameter d i gives the location of the item on the latent continuum; t ijdenotes an additional step parameter.ACER Conquest, Version 2.0 software (Wu, Adams, Wilson, & Haldane, 2007) was used toscale the ICCS test data.129