ICCS 2009 Technical Report - IEA

Chapter 9:Quality assurance in the ICCS datacollectionBarbara Malak and Alana YuIntroductionThe ICCS international research team expended considerable effort on developing standardizedmaterials and operational procedures for ICCS (see Chapter 8 for more information) so thatthe data collected in each country would be comparable to the greatest possible extent. Qualitycontrol encompassed internal mechanisms built into each stage of the data-collection processto ensure the quality of the ICCS data. Quality control also encompassed external reviewsadministered by trained quality control monitors (QCMs), individuals who were separate fromthe staff being evaluated. Quality control of the data collection was an integral part of ICCS atboth the national and international levels.Two independent quality-assurance programs were implemented in each participating country:an international quality-assurance program conducted by the IEA Secretariat, and a nationalquality-assurance program managed by each national center (both programs employed similarquality-control measures). The main purpose of these programs was to provide documentedevidence that the countries followed standard international data-collection procedures at alltimes. This chapter describes the implementation and outcomes of these programs.The IEA Secretariat appointed international quality control monitors (IQCMs) for each participatingcountry. The NRCs in each country nominated suitable people for this role and submittedtheir recommendation to the Secretariat. The IQCMs observed the data-collection session ina random sample of 15 schools per country, and interviewed the people responsible for thisactivity. Altogether, 98 IQCMs and their assistants observed 535 testing sessions in 37 of the38 countries that participated in ICCS. 1 They conducted their observations and interviewsaccording to a defined protocol, and documented their observations and the interviewresponses in a standard form.Another item of documentation was the survey activities questionnaire for the NRCs. They wereasked to relate their experiences during implementation of the ICCS survey procedures andassessment materials. They also provided information about their national quality-assuranceprograms.Quality control observations of the ICCS data collectionInternational quality-assurance programThe program was carried out by the IEA Secretariat. A core element of this program was theappointment of one or more International Quality Control Monitors (IQCMs) in each country,each of whom had been nominated by the respective national centers. The IQCM had tobe someone external to the national center and familiar with the school environment (e.g.,a school inspector, ministry official, retired school teacher), fluent in both English and thelanguage(s) spoken in the schools to be visited, and likely to be acceptable as an observer at theselected schools. Where necessary, the IQCMs could recruit one assistant or more in order toefficiently cover the territory and testing timetable.All monitors participated in a training seminar conducted by the IEA Secretariat to preparethem for completing the tasks associated with the international quality-assurance program.1 IQCM reports were not received from the Dominican Republic.103