ICCS 2009 Technical Report - IEA

As was the case with school coordinator training, most countries used more than one approachto train their test administrators. The main reliance was on written instructions (32 countries)supplemented with formal training sessions (27 countries) or email/telephone correspondence(13 countries). A number of NRCs reported difficulties reaching high participation rates forstudents (9 countries), teachers (18 countries), and high school principals (9 countries).Some schools had problems with truancy, and logistical issues meant that it was not alwayspossible to schedule a make-up session for absent students. NRCs reported lack of time, interest,and/or willingness to answer questions about political topics as the reasons that teachers andprincipals most frequently gave for non-response.National quality control monitoringEach national center received materials to help them conduct their own national qualityassurance programs. The ICCS 2009 Survey Operations Procedures, Unit 2 provided guidelines forselecting the national quality control monitors (NQCMs), as well as basic information about themonitors’ duties. The ICCS 2009 National Quality Control Monitor Manual (ICCS ISC, 2008j) wasdeveloped to assist training. The ISC encouraged national centers to amend the manual and theprovided classroom observation record so that these reflected any matters of special importancein their respective countries. The NQCMs’ responsibilities were similar to those of the IQCMs,in that they involved visiting selected schools to observe and document the data-collectionsession.According to the information given by the NRCs in the survey activities questionnaire, a totalof 191 NQCMs and their assistants visited 536 schools in the participating countries. TheNQCMs confirmed the good quality of the surveying process overall. Like the IQCMs, theydetected several problems, such as defective survey materials, test administrators making errors,and unmotivated students. However, the issues that they reported were generally minor andwere resolved promptly and appropriately (e.g., calling schools to verify the correct shipment ofinstruments, rescheduling the survey due to a teachers’ strike, and shifting the European moduleto another day to cope with test fatigue among students). One school prematurely opened abox of questionnaires, which meant that this school’s data were not included in the database. Asevident in Table 9.14, only one country (Slovenia) did not use the provided support materialsbecause it did not implement a national quality assurance program. (Slovenian national centerstaff conducted the test administration.)Table 9.14: Numbers of NRC responses to the survey activities questionnaire: national quality control monitoringQuestion Yes No Not AnsweredDid you use the national quality control monitor templates (manual and 37 1 0classroom observation record) provided by the international study center?Scoring open-ended response items and coding occupation dataNRCs were also asked to comment on the persons responsible for scoring open-ended responseitems, the procedures used to verify scoring reliability, and the coding of occupation data(see Table 9.15). Between them, the participating countries used 349 scorers from a varietyof backgrounds. The scorers typically included national center staff, teachers, and universitystudents. Six NRCs said they experienced difficulties with implementing the reliability(independent double scoring) procedure in their countries. However, these difficulties generallyoccurred early in the process and were resolved after the scorers experienced additional scoringpractice.QUALITY ASSURANCE IN THE ICCS DATA COLLECTION117