ICCS 2009 Technical Report - IEA

The ICCS test of civic knowledge included a link to the CIVED survey in 1999 through theinclusion of secure items from the CIVED item pool. The inclusion of these 17 multiple-choiceitems made it possible to measure changes in performance for countries that participated inboth ICCS and CIVED and had assessed comparable student populations.Test-development processThe cognitive test-item and instrument development process consisted of a series of stages.These stages followed one another sequentially. However, the iterative and collaborative natureof the overall process meant that some materials were reviewed and revised within particularstages more than once.NRC test-item development workshopThe item development process began formally with an item development workshop at the firstmeeting of NRCs in October 2006 in Amsterdam. At this workshop, national representativeswere provided with information about the framework and procedures for ICCS testdevelopment. Participants drafted items in small working groups. The workshop involved thefollowing activities:• A review of the content of the assessment framework to ensure a common understandingof the fundamental civics and citizenship constructs;• A mapping of the CIVED trend items against the assessment framework to guide thedevelopment of new items;• Confirmation of the necessary properties of test-item stimuli, including issues relating toensuring cultural sensitivity and avoiding potential biases (such as cultural or gender bias);• Confirmation of item types, scoring-guide formats, and test-development systems,including the online item-review process;• An introduction to the principles of cognitive test-item development;• The opportunity to discuss and consider any cognitive test items that NRCs had broughtto the workshop;• Test-item development in small groups; and• An invitation for NRCs to make arrangements for cognitive test items to be developed andsubmitted to the ISC for consideration.Stimulus selection and preliminary item developmentThe focus of this preliminary stage of item development was on establishing authentic, viable,and relevant contexts for items to assess the content specified in the assessment framework.Stimulus materials, contexts, and item ideas were developed internally at the ISC. NRCs andmembers of the PAC submitted a small number of contexts and ideas, some of which had beencreated at the item-development workshop.These materials were submitted to the ISC, where the test-development team assessed themfor their suitability for further development. This work included evaluating the extent towhich these materials were appropriate for civic and citizenship assessment purposes and theirsuitability for the target student population. The team also reviewed these materials with respectto the range of contexts and themes that they covered.Materials selected for further development were subsequently refined (as required), and therelated items were developed by a test developer responsible for a particular unit (stimulus,items, and scoring guides). Once the project researchers had developed their respective units tothe degree that they considered them to be complete, they submitted them to a quality controlprocess called “paneling.”ICCS test development23