IEA Civic Education Study. Technical Report. Amsterdam - ACER

THE PROCESS OF TEST AND SURVEY DEVELOPMENT FOR14-YEAR-OLD STUDENTSBecause there were no large existing sets of items likely to yield the number ofitems needed to fill in the matrix, extensive item writing was required. Webegan by reviewing materials in the content guidelines, other summaries ofPhase 1 documents, and messages exchanged during an online conference oncivic issues conducted with secondary school students in seven countries. Wenext invited all NRCs to submit items. Our third task was to review the 1971CivEd instrument, released items from the United States and Canadianassessments, and the published research literature. Members of the InternationalSteering Committee (ICC) then wrote items, which were subsequently enteredinto an item database keyed to the content guidelines. Our fifth step involvedasking groups of test specialists and content experts to review items in thedatabase and their relation to the content framework. All NRCs whosecountries were officially participating by September 1997 reviewed itemsbefore the pre-pilot testing; all NRCs whose countries were officiallyparticipating by November 1998 reviewed items before their inclusion in thefinal version of the test.The result of this activity was the development of 140 knowledge and skillsitems (content knowledge and skills in interpretation), each with one correct answerand four distracters, and each of which was entered into the database for the14-year-old population. All the items were judged suitable for administration inthe participating countries.The items focused on principles, pivotal ideas, and general examples, and noton the details of the political arrangements in any one country. For example,Content Knowledge/Domain I items covered the principles of democracy andits associated institutions across the countries participating in the study. Thetest did not include items about specific mechanisms of the electoral process orgovernment structure in any particular country. The Content Knowledge/Domains II and III items likewise dealt with internationally relevant orgeneralized matters shared across countries. This emphasis differs from that inmany national tests where items about each country’s political structurepredominate. The CivEd Phase 2 items are based on information gatheredduring Phase 1 about what students are expected to know and on expertstatements such as that issued under the auspices of the Council of Europeabout the role of history knowledge in civic education (Slater, 1995, pp. 146–148).Some of the skills in interpretation items asked students to distinguish betweenstatements of fact and opinion. Others were based on a leaflet of the typeissued during an election campaign, on the interpretation of a short article froma mock newspaper, or on a political cartoon. The general ideas for cartoonscame from those published in newspapers. They were redrawn to communicatea single message that 14-year-olds across countries could be expected tounderstand.20 IEA CIVIC EDUCATION STUDY TECHNICAL REPORT