ICCS 2009 Technical Report - IEA

Chapter 13:The reporting of ICCS resultsWolfram SchulzIntroductionThis chapter describes the procedures that were used to report results in the ICCS publications.The chapter begins with a description of the replication methodology used to estimatesampling variance and then provides an outline of how the imputation variance of the civicknowledge scores was computed. The subsequent section describes how the significance testsfor differences between country and subsample means or percentages were conducted.This chapter also includes descriptions of how the multiple regression analyses were conductedand how the hierarchical models explaining civic knowledge were estimated. The final sectionof the chapter outlines how missing data were treated during multivariate analyses of the ICCSdata.Estimation of sampling varianceICCS employed two-stage cluster sampling procedures to obtain the student as well as theteacher samples. During the first stage, schools were sampled from a sampling frame with aprobability proportional to their size. During the second stage, intact classrooms were randomlysampled within schools. Cluster sampling techniques permit an efficient and economic datacollection. However, because these samples are not simple random samples, the usual formulaeused to obtain standard errors for population estimates are not appropriate.Replication techniques provide tools with which to estimate the correct sampling variance onpopulation estimates (Gonzalez & Foy, 2000; Wolter, 1985). ICCS used the jackknife repeatedreplication technique (JRR) to compute standard errors for population means, percentages,regression coefficients, and any other population statistic.Generally, the JRR method for stratified samples requires pairing primary sampling units(PSUs)—in this survey, schools—into pseudo-strata. Because assignment of schools to these“sampling zones” needed to be consistent with the sampling frame from which they weresampled, sampling zones were constructed within explicit strata. Occurrences of an oddnumber of schools within an explicit stratum or the sampling frame saw the remaining schoolrandomly divided into two halves, thereby forming a sampling zone of two “quasi-schools.”Each of the countries participating in ICCS had up to 75 sampling zones. In countries witha large number of participating schools, some schools were combined into bigger “pseudoschools”in order to keep the total number to 75. A three-stage sample design was appliedto the Russian Federation. The first stage of this process consisted of a sample of regions. Ifa selected region was large enough to be selected with certainty, schools were paired. If thiswas not the case, regions in the sampling zones were paired. In countries where all schoolswere tested and two classrooms within each school had been sampled (i.e., Cyprus and Malta),schools were defined as sampling zones and classrooms as PSUs. In countries with censussurveys (Liechtenstein and Luxembourg), students were randomly assigned to sampling zonesand quasi-schools. The same procedure was applied to teachers in countries in which all schoolswere selected. Table 13.1 shows the range of sampling zones for the student, school, andteacher samples used in each participating country.Within each of the sampling zones, one school was randomly assigned a value of 2 and theother school a value of 0. Replicate weights were computed for each of the sampling zones.This meant that one of the paired schools had a contribution of zero, the second a doublecontribution, and all other schools remained the same. The replicate weights procedure isachieved by simply multiplying student or teacher weights with the jackknife indicators onceonly for each sampling zone.261