ICCS 2009 Technical Report - IEA

For countries administering the school and teacher questionnaires online, instrumentpreparation comprised an additional verification step. Countries were asked not to set uptheir online questionnaires until the paper-based instruments had been verified (as describedin Chapter 5). Countries then used the IEA SurveySystem to set up the survey online. Toensure that data from both administration modes were comparable, the IEA DPC conducted asystematic check of the paper and online questionnaires. Apart from a few inevitable exceptions,which were necessary because of the different administration modes and which were set downfor NRCs in “online adaptation notes,” 3 any deviations with regard to content and layoutbetween paper and online instruments were reported back to the countries. In such cases, NRCswere requested to update their online instruments to match the paper instruments.Administering the ICCS assessmentDistribution of the printed materials to the schools required the national centers to engage incareful organization and planning. The labels and student tracking form produced by WinW3Sallowed each sampled student to be assigned one test booklet. The test booklets were assignedin a completely balanced rotated design so that each test-item cluster within the booklets wasassigned to an approximately equal number of students. Each student was also assigned a studentquestionnaire, labeled in a way that linked it to the corresponding test booklet. Depending onthe country’s participation in a regional module, each student was also assigned a regionalinstrument. The materials were packaged separately for each sampled class. In addition, theteacher questionnaires were assigned and sent to each school for each teacher listed on the teachertracking form. A school questionnaire was sent to the school principal.For teachers and school principals who would be completing their questionnaires online,national centers prepared and sent cover letters that contained login information andinstructions on how to complete the online questionnaire. National center staff sent thepackaged materials to the school coordinators prior to the testing date and asked themto confirm the receipt of all instruments. School coordinators then distributed the schoolquestionnaire and teacher questionnaires (or the cover letters for the online participants) whileensuring that the other instruments were kept in a secure room until the assessment date.Having referred to the relevant procedures described in the Test Administrator Manual, nationalcenters assigned a test administrator to each sampled class. This person’s role was to administerthe test along with the student questionnaires and a regional instrument (where applicable). Thisperson was chosen and trained by the national center, although, in some cases, the schoolcoordinator also filled the test administrator role. The test administrator was responsible fordistributing materials to the appropriate students, leading students through the assessment, andaccurately timing the sessions. After students had completed the test, the test administrators alsoadministered the student questionnaire as well as a regional instrument (where applicable).The administration of the ICCS assessment consisted of either two or three parts. The firstpart concerned the achievement booklets. This was followed by the completion of the studentquestionnaire. If the country participated in a regional module, a regional instrument wasadministered afterwards. The time allotted for each part was standardized across countries.To complete each part of the achievement test, target-grade students were allowed 45 minutes.If a student had completed the assessment before the allotted time was over, he or she couldreview his or her answers or read quietly but was not allowed to leave the testing room. Inorder to complete the student questionnaire, students were given at least 40 minutes, and were3 For example, in online questionnaires, respondents were redirected automatically by way of filtering rules. In contrast,the paper questionnaires required respondents to turn pages manually. Hence, instructions differed depending on theadministration mode.94 ICCS 2009 technical report