IEA Civic Education Study. Technical Report. Amsterdam - ACER

Piloting of Items Assessing Content Knowledge and Skills inInterpretation and the Resulting Final TestBetween April and October 1998, 25 countries conducted pilot studies on thetwo forms of the test (items assessing content knowledge and skills in interpretation,described above) and the survey (items assessing concepts, attitudes and actions,described below). In each country, judgment samples of about 200 studentswere tested (two class periods per student). The pilot countries includedAustralia, Belgium (French), Bulgaria, Chile, Chinese Taipei, Colombia, Cyprus,Czech Republic, Estonia, Finland, Germany, Greece, Hong Kong (SAR),Hungary, Italy, Latvia, Lithuania, Norway, Poland, Portugal, Romania, theRussian Federation, Slovenia, Switzerland, and the United States. In addition tothese countries, Denmark, England, the Slovak Republic, and Swedenparticipated in the final testing of 14-year-old students. (Chinese Taipei wasunable to obtain funding to continue its participation after the pilot testing.)The NRCs were provided with item statistics for their countries (difficulty,discrimination, and differential item functioning). They discussed each itemwithin its content category at a November 1998 meeting. The small number ofitems that was viewed as unacceptable by more than one-fifth of the NRCs wasdropped in accordance with the rule used by IEA to promote test fairnessacross countries. Through a process of negotiation, the research coordinatorschose, by consensus, 38 items assessing content knowledge and skills ininterpretation from the 68 that had been piloted. The discrimination indiceswere greater than .30 for most of these items. In addition to psychometricproperties, coverage of the content framework and the research coordinators’preferences were decisive factors in the process of item selection (see AppendixB for a listing of items by content category).The ratios of number of “items written” to “number piloted” to “numberaccepted” were similar to IEA tests in other subject areas. Confirmatory factoranalysis and IRT modeling indicated a high-quality test across countries. Thesemodern scaling methods (Frederikson, Mislevy, & Bejar, 1993) were ourprimary guide as we developed the test. Classical scaling methods alsoindicated a test of high quality. The alpha reliabilities for the final 38-item civiceducation test exceeded .85 in each of the countries.Piloting of Items Assessing Concepts, Attitudes, and Actions andthe Resulting Final SurveyAt the March 1998 meeting, the NRCs reviewed lists of suggested topics foritems assessing concepts, attitudes and actions and some prototype items. Mostitem sets for piloting were suggested by the research literature. Some revisionswere necessary to adapt items originally designed for administration to adultsin an interview, and a “don’t know” option was added to each of these items.In mid-1998, the research coordinators piloted the survey items along withtwo forms of the knowledge and skills test. Items for the survey were chosenthrough a process of negotiation similar to that described in the previoussection. The final survey included 52 items assessing concepts (concepts ofdemocracy, citizenship, and government), 62 items assessing attitudes (social and22 IEA CIVIC EDUCATION STUDY TECHNICAL REPORT