ICCS 2009 Technical Report - IEA

The sampling variance SV mis calculated as the average of the sampling variance for eachplausible value SV P m :PSV µ = P1 S SVPp=1 m .Use of the P plausible values for data analysis also allows one to estimate the amount of errorassociated with the measurement of civic knowledge. The measurement variance or imputationvariance IV p is computed asPIV p =1P-1 S (µp–µ p=1 q q )2.Here, µ p is the statistic of interest computed on each plausible value p and µ is the meanq qstatistic based on all P plausible values.The estimate of the total variance TVµ, consisting of sampling variance and imputationvariance, can be computed asTV µ = SV µ + (1+ P 1 )IV µ .The estimate of the final standard error SE µ is equal toSE µ = TV µ.Table 13.3 shows the average scale scores as well as their sampling and overall standard errors.It also records the number of students that were assessed in each country. The comparisonbetween sampling and combined standard error shows that most of the error was due tosampling and that only a small proportion could be attributed to measurement error.Use of the IEA IDB Analyzer for estimating civic knowledge standard errors allowed bothsampling and imputation errors to be automatically calculated and combined. Alternativeapplications, such as the software package WESVAR (Westat, 2007) or the ACER SPSSReplicates Module, have similar features that make it possible to take the imputation error intoaccount when analyzing the plausible values.Reporting of differencesSignificance tests were conducted for:• Differences in population estimates between countries (multiple comparisons);• Differences between a country and the international average;• Differences in population estimates between subgroups within countries; and• Differences between population estimates in ICCS and in CIVED (estimation of changes).Differences between two score averages (or percentages) a and b were considered significant(p < 0.05) when the test statistic t was greater than the critical value, 1.96. The calculation of twas conducted by dividing the difference by its standard error, SE dif_ab :t = (a-b)SE dif_ab .In the case of differences between score averages from independent samples (evident, forexample, with respect to comparisons of country averages), the standard error of the differenceSE dif_ab can be computed as:SE dif_ab = SE 2 a + SE 2 b.Here, SE a and SE b are the standard errors of the means from the two independent samples a and b.SCALING PROCEDURES FOR ICCS questionnaire ITEMS265