University of Oslo Workshops June 29-30 Conference July 1-3 ...

The Impact of Missing Data on Subgroup Estimation
Leslie Rutkowski, Indian Universiy, USA
TIMSS and PIRLS use a sophisticated assessment design whereby each individual student
is only administered a small number of the total possible cognitive items. In particular
TIMSS and PIRLS use a rotated booklet design that ensures that each cognitive item
receives sufficient exposure and that each examinee receives a sufficient number of items
to estimate population-level achievement in a number of domains and sub-domains. While
this method of item delivery is efficient from an administration perspective, the approach
poses intractable challenges for precisely estimating individual student achievement. To
overcome the methodological challenges associated with multiple-matrix sampling,
international LSA programs adopted plausible value methods that use marginal estimation
techniques to generate population level achievement estimates (Mislevy, 1991; Mislevy,
Beaton, Kaplan & Sheehan, 1992; Mislevy, Johnson & Muraki, 1992). While the plausible
value methodology has been well established theoretically and empirically, a paucity of
literature appears to exist regarding the effect of poor quality background data on
subpopulation achievement estimates. In the current paper, we seek to examine the impact
of missing background data used to estimate subpopulation achievement. First, we reason
that group differences will be underestimated when background variables used to optimize
subpopulation achievement differences have high rates of missing (at random) data.
Second, we examine the impact of background data that are systematically missing or
missing not at random. Data are simulated using TIMSS 2007 item parameters and a
variety of conditions.
Keywords: TIMSS; plausible values; subgroup achievement estimation
On the Degree of Comparability in Trend Studies as a Function of
Differences in Age and Schooling
Monica Rosén, University of Gothenburg, Sweden
Rolf Strietholt, Dortmund University of Technology, Germany
Linking IEA-studies on reading literacy of 9-10 year-old students via IRT-technique
provides an extensive source for trend analyses from 1970 to 2006. Comparison derives
from the assumption of having comparable groups in respective studies and countries.
Regarding this, students’ age and time of schooling play a crucial role since the subsamples
cover students that differ in terms of age, grade and test months. Therefore, the
comparability can be considered as a function of differences in age and schooling. The
main purpose of our analysis is highlight issues of study design that appear in trend
analysis and affect the comparability. We discuss miscellaneous strategies to handle such
differences and address limitations of these approaches.
Keywords: age effect; comparative study; reading literacy; schooling effect
56