ICCS 2009 Technical Report - IEA

Chapter 7:Sampling weights and participationratesOlaf Zuehlke and Caroline VandenplasIntroductionA major objective of ICCS was to obtain accurate, precise, and internationally comparableestimates of population characteristics. Several considerations had to be taken into account toachieve this goal.This chapter begins with an outline of the definition of what constituted student or teacherparticipation and what constituted the requirement for within-school participation withineach sampled school. Not every student or teacher who completed a survey instrument wasautomatically regarded as a participant in ICCS. Also, because the risk of bias greatly increasesif only a minority of the sampled students or teachers in a school participate in the survey, datafrom affected schools were disregarded.The next three sections of the chapter contain a description of the several sets of weightsthat were computed to ensure results based on ICCS data resembled those in the underlyingtarget populations. As explained in Chapter 6, the complex sampling design of ICCS resultedin varying selection probabilities for the selected students and teachers. Furthermore, varyingpatterns of non-participation between strata had the potential to bias results. Both factorsemphasized the need to use weighted data to achieve accurate estimates of populationparameters. To this end, the IEA Data Processing and Research Center (DPC) calculatedweights for all participating units in ICCS. All findings presented in ICCS reports are based onweighted data. Anyone conducting secondary analysis of the data in the ICCS database shouldfollow this approach.The final section of this chapter describes the participation rates at each sampling stage, theminimum acceptable participation requirements (unweighted and weighted) for students andteachers, and the categories of sample implementation quality that each country achieved. TheICCS research team regarded response rates as an important indicator of data quality. Althoughthe team made considerable effort to ensure full participation, not all sampled units wereincluded in the study. National samples were accordingly adjudicated with regard to sampleparticipation requirements in the student and teacher surveys.Within-school participation requirementsStudent survey participation requirementsWhen the student response rate within a school is very low, the likelihood of biased resultsincreases. There is evidence that low-performing students in particular tend to be morefrequently absent from school than high-performing students. Therefore, ICCS defined arequired minimum student participation rate within each school. This rate determined whetheror not a school could be considered a “participant” in ICCS.In most participating countries, only one class per school was selected for ICCS. In thesecountries, schools had to meet the following participation requirement:• A sampled school was regarded as a “participating school” if, in its sampled class, at least50 percent of its students participated in the student survey.• If a school did not meet this requirement, it was regarded as a non-participating schoolin the student survey. The non-participation of this school had an effect on the schoolparticipation rate, but the students from this school were not included in the calculation ofthe overall student participation rate.69