ICCS 2009 Technical Report - IEA

As a consequence, and after effort had been made to control for ability, the CIVED studentshad a higher probability than the ICCS students of giving correct answers to items that hadbeen administered at the beginning. However, the CIVED students were less likely to knowthe answers to test questions administered toward the end of the assessment. This effect was,however, not very strong for the pooled international samples and was notable in only a fewcountries.Given that the framework for the test domain was broader in ICCS than in CIVID and giventhat the majority of the link items represented only one content domain (civic systems andsociety), it was not surprising to find in the preliminary analysis some notable differences in thebehavior of the CIVED link items and the new ICCS items. Effort to estimate comparable ICCStest scores using the newly established scale for the CIVED data was not deemed appropriate.A decision was made to test an approach that involved using different equating methods to settest data for the link items (with reduced sample size, given these items appeared in only threeout of the seven ICCS booklets) against the CIVED scale metric. However, comparisons of thedifferences in percentages correct for both surveys and the resulting trend estimates showedseveral inconsistencies that were probably a consequence of the set of link items including onlytwo items from the subdimension “interpretative skills.” It was therefore decided to report onlychanges pertaining to the “civic content knowledge” subscale, for which 15 link items wereavailable.In order to review the link item characteristics, the adjusted item difficulty parameters (eachstandardized to have a mean of 0) were compared first at the international level and then foreach national sample. Figure 11.6 shows the scatterplot between the item parameters fromCIVED and those estimated for the trend sample consisting of 500 randomly selected studentsfrom each of the national samples with comparable data. As is apparent in the figure, five itemswere slightly outside the error bands derived from the respective standard errors of the itemparameters from both calibrations. However, the figure also shows that the item parameterswere generally highly similar; the correlation between item parameters was 0.96.As was the case for the international scaling, some national items had to be omitted fromscaling. These items included those that were excluded from the CIVED scaling (see Schulz& Sibberns, 2004), ICCS versions of items with translation errors or deviations, items thatreflected printing problems, and items that showed very large differences in relative itemdifficulty between the two surveys. Table 11.11 shows the national items that were excludedfrom the equating procedures for ICCS data as well as the reasons for their exclusion.For the final scaling, maximum likelihood estimates (MLEs) were derived using the same itemparameters as in CIVED. They were then transformed to the same scale metric, which wasset to have a mean of 100 and a standard deviation of 20 for the 28 CIVED countries. Thetransformation of scale scores was computed by applying this formula:q n ’=100+20 q n–qs q .Here, q n ’ are the student scores in the CIVED metric, q n are the original logit scores (maximumlikelihood estimates), q is the CIVED mean of student logit scores (0.95) with equally weightedcountry subsamples, and s qis its corresponding CIVED standard deviation (1.36). Table11.12 shows the item parameters used for scaling as well as the average percentages of correctresponses for these items in the 17 countries in 2009 and 1999.SCALING PROCEDURES FOR ICCS TEST ITEMS149