ICCS 2009 Technical Report - IEA

Table 9.3: Percentages of IQCM responses for administration of the ICCS student questionnaireQuestion Yes (%) No (%) Not Answered (%)Did the test administrator follow the instruction exactly in each of Minor changes Major changesthe following tasks?• Preparing the students 84 6 4 6• Distributing the materials 89 4 2 5• Giving directions 77 13 4 6• Giving examples 81 6 5 7If the test administrator made changes to the instruction, howNot answered Not applicablewould you describe them?• Additions 12 8 4 76• Revisions 8 11 6 76• Deletions 9 9 7 76Were students asked after 40 minutes if they had all completed 69 26 5the questionnaire?Was additional time allowed? 37 58 5Were the questionnaires collected and secured after the 92 3 6questionnaire session?Summary observationsThe IQCMs provided, in Section C of the classroom observation record, their generalimpressions of how the testing session was conducted, how well the test administratormonitored students, and any unusual circumstances that arose during the session (e.g., studentsrefusing to participate, defective instruments, cheating). The results presented in Tables 9.5 and9.6 show that, for most testing sessions, the IQCMs observed no major problems.Table 9.5 reports the IQCMs’ general observations of student behavior and the quality of theadministration session. In nearly all instances, the IQCMs considered the students orderly andcooperative. Ninety-one percent of the IQCMs described the overall quality of the observedsessions as “excellent,” “very good,” or “good.”Table 9.6 presents various other observations made by the IQCMs. Occasionally, they notedthat while space constraints in the survey rooms prevented the test administrators from walkingaround the class, they were still able to monitor students from the front of the room. In almostall cases, the IQCMs considered that the test administrators had addressed students’ questionsappropriately. In 15 percent of cases, IQCMs reported evidence of students attemptingto cheat. However, in many of these instances, the IQCMs characterized the situation as“communicating” rather than cheating, explaining that students seemed curious about how theirclassmates had responded to items in the questionnaire. They also said that test administratorsintervened when necessary. Because the ICCS test design involved seven different achievementbooklets, students were unlikely to have had the same booklet as their neighbors. In thefew sessions where defective instruments were detected, the test administrator almost alwaysreplaced the booklet appropriately.There were very few reports of students refusing to take the survey, and when this did occur,the students were typically observed to have ended the survey early (due, for instance, to aprior appointment, illness, or presumed lack of interest in continuing the assessment). Morecommonly (in 18% of observations), IQCMs reported that students briefly left the room duringthe session. On nearly all of these occasions, the test administrators responded appropriately bycollecting the booklet; in some sessions, the booklet was left closed on the student’s desk untilthe student returned to class.108 ICCS 2009 technical report