ICCS 2009 Technical Report - IEA

Scoring• Was the proposed scoring consistent with the underlying ability measured by the test andwould test respondents with higher ability levels always score better than those with lowerones?• Were there different possible student responses that might receive the same score, and didthese responses represent equivalent or different levels of proficiency?• Were there other kinds of answers that had not been anticipated in the marking guide(e.g., any that did not fall within the “correct” answer category description but appeared tobe equally correct)?• Were the scoring criteria sufficiently clear for coders to allow them to distinguish thedifferent levels of performance?The reviews and evaluations conducted during the paneling process provided the participatingtest developers with extensive notes on each stimulus piece, item, and scoring guide (for theconstructed-response items). The item material deemed appropriate for further development wassubsequently refined on the basis of the panel’s feedback.Refinement of item materialDuring the process of refinement, all revised materials were shown to at least one test developerwho had not previously seen them. The purpose of this additional check was to ensure that therevision of items had not created additional technical problems.External reviewAll ICCS draft test material (stimulus items and scoring guides) were placed on the ICCSwebsite for review by members of the PAC, NRCs, and other consultant experts. The webbasedreview process allowed reviewers to post comments on each component of the materialsand to complete a brief rating (1 to 4) of the suitability of the materials for inclusion in the test.The rating categories were:4: Include item without change;3: Include item, changes recommended;2: Include item only if changes have been made;1: Do not include item.The web-based external review took place in March and April 2007, after which the test draftmaterial was further revised in accordance with the feedback arising out of the review.PilotingThe draft test items were piloted at the same time as the web-based item review. Two short testforms, each with 42 items, were created using the draft item material and delivered to smallerconvenience samples of students in Australia, Colombia, England, Italy, the Netherlands, andNew Zealand. Overall, 436 students participated in the pilot study. In addition to completingthe test items, Australian students participated in small group discussions that centered ontheir experiences when completing the piloted test items. Students were asked to commenton their perceptions of the difficulty of the items, the wording of the items (with a focus onany ambiguities or difficulty they had in terms of understanding each item), and the content(context as well as the civics and citizenship content) of the items.The results from the pilot study provided a first empirical basis for further development ofthe ICCS test material. The student responses to the open-ended test items in the pilot wereparticularly useful for informing the further development of both items and scoring guides.ICCS test development25