12.07.2015 Views

30. Teacher evaluation of reading Level 4 assessments ... - e-asTTle

30. Teacher evaluation of reading Level 4 assessments ... - e-asTTle

30. Teacher evaluation of reading Level 4 assessments ... - e-asTTle

SHOW MORE
SHOW LESS

You also want an ePaper? Increase the reach of your titles

YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.

Technical Report 30 2002Teacher Evaluation of the Reading Level 4 Assessments: SummaryTechnical Report 30, Project asTTle, University of Auckland, 2002Jeremy B. Zwiegelaar & Gavin T. L. BrownUniversity of AucklandThis report summarises feedback responses of the teachers (n=20) involved in theevaluation of asTTle reading level 4 papers in November 2001. The majority of teacherand student responses to the reading assessments were positive, though more negativecomment is expressed from students regarding the difficulty of items.Table of ContentsQuestion 1 .........................................................1Question 2 .........................................................2Question 3 .........................................................2Question 4 .........................................................2Question 5 .........................................................2Question 6 .........................................................3Concluding Comment .......................................3A trial of new asTTle reading assessmentswas conducted in November 2001. The trialconsisted of three reading papers namely I, J,and K each of which was designed to assessclose reading at Level 4 Advanced of the NewZealand English curriculum. New items andtexts were written, and analysed (See Brown,2002, Technical Report 28 for details of theitems and texts) to meet the need for greaternumbers of difficult Level 4 reading items.Because of the difficulty of the questions andpassages it was decided to calibrate the items onstudents in Years 7, 8, and 9 only to ensure thatthe items were within the capability of studentsin the target range. However, given thewidespread of abilities in mainstream classes itwas anticipated that some students would findthe items challenging. The new items and taskswere linked with selected Level 4 Proficient andAdvanced materials already calibrated inprevious testing.Nine schools, of which seven agreed toparticipate, were invited to supply three classesof students per target year level. The schoolswere in the Auckland region and represented arange of socio-economic strata. Just over 950scripts were returned from a total of 1,150 sentout. Each teacher who administered papers to aclass of students was asked to complete afeedback form. The potential pool ofrespondents, based on one teacher per class,would be 36. A total of 20 replies werereceived. This response sample represented justover half of all teachers potentially involved inadministering the asTTle papers.Responses were in the nature of commentsto prepared questions. The comments weregenerally coded using a “Yes”, “No”, “Both”,or “No answer”. The category “Yes” indicatesa favourable or positive response to thequestion, a “No” indicates an unfavourable ornegative response to the question, and “Both”indicates a response that contains both positiveand negative comments. “No answer” includescomments that were incapable of meaningfulinterpretation. Because teachers administeredmore than one paper, responses are summarizedby type of paper rather than by individual paper.Question 1Was the content appropriate for the age leveland ability of the students?More than half of the teachers were satisfiedwith the content of the reading papers (Table 1).Only one teacher was dissatisfied and one-fifthof teachers considered the content neitherappropriate nor inappropriate for the age andability level of students.Table 1Appropriateness of contentYes No Both No TotalanswerN 13 2 4 1 20Percentage 65 10 20 5 1001


Zwiegelaar, J. B., & Brown, G. T. L.Comments indicating a favourable responseincluded: “Yes. Students could relate to it andunderstand the information”. Negativeresponses included “Some words were quitehard to understand and some questions weretoo hard (student responses)”.Question 2Was the content interesting and engaging forstudents?There was strong support for the interestlevel and degree of engagement the testsprovided, with only one teacher totallydisagreeing (Table 2). The content wasconsidered interesting and engaging for studentsas suggested in one response: “Yes- theredefinitely was a variety of types of questions aswell as texts”.Table 2Interest & engagement of contentYes No Both No TotalanswerN 13 2 4 1 20% 65 10 20 5 100While it was the case that the overwhelmingmajority was happy with the material’s level ofinterest, the answers that were coded as ‘Both’were often comments relating to the fact thatdue to differing interests, the tests did not suitall of the children in the class, which was to beexpected.One of the negative responses related to thedifficulty of one standard test fitting allstudents’ interests. “No, It was not somethingthat stimulates the mind. To keep concentrationgot difficult at times.”Question 3Were the teacher’s instructions clear, easy tofollow and sufficient?There was considerable support for the testsin terms of their clarity of instructions (Table3). However, there were still concernsexpressed, and suggestions offered for ways inwhich to improve the papers. Suggestionsincluded “The teacher instructions would havebeen more useful if the practice questions hadbeen outlined with the instructions”. One of thenegative responses was that: “No. There was nobox to record the school number. The practicequestion instructions were not specific e.g.,‘check that students know how to answer…’P.A.T. type verbal instructions and numberingwould be helpful”.Table 3Clarity & ease of teacher’s instructionsYes No Suggestion Both No TotalmadeanswerN 14 1 3 1 1 20% 70 5 15 5 5 100Question 4Was the level of difficulty appropriate forProficient to Advanced Level 4 students?More than half the teachers believed thedifficulty of the papers was appropriate for thestudents working at proficient to advancedlevels (Table 4). Two teachers were negativeabout the appropriateness for these levels ofdifficulty, though their comments that the testswere “more to the advanced end of thespectrum” were consistent with the design briefof the materials. One teacher remarked thathe/she felt unable to comment until the paperswere marked and that he/she had seen theresults.Table 4Appropriateness of difficulty across the papersYes No Both No answer TotalN 12 2 3 3 68% 60 10 15 15 100Question 5What was the response of the students to thepapers?The majority of respondents made a generalcomment (Table 5). Despite the fact that theresponses were wide and varied, they fell intofive main categories – positive, negative, both,uninterpretable, and suggestions given. Therewere slightly more negative comments thanpositive ones made by the students. Thenegative comments identified the issue of thetest materials being inappropriate for certainclasses or types of students (i.e., too long, toohard, too easy). This was to be expected giventhat the test materials were designed forProficient and Advanced Level 4 and that


Teacher Evaluation Reading Level 4 3classes of students contain students rangingfrom Levels 2 to 5 of the English curriculum.Table 5Student responses to paperComment N PercentagePositive 7 35Liked/enjoyed/interesting 4 20No worries 2 10Good/positive 1 5Negative 9 45Boring 3 15Hard 4 20Too long/too easy/finished 2 10earlyNeutral/Mixed 2 10Liked some parts and disliked 1 5othersSuggestions for improvement 1 5Response unclear 1 5No answer 1 5Totals 20 100Question 6Are there any other general comments youwould like to make?All but 9 responses recorded some commenton behalf of or in summary of student responseto the papers. A majority of the teachers whocommented reported that students enjoyed theexperience (Table 6), while only three teachersreported students not being positive to thepaper.the need to separate desks etc. I would like toretain this feature if possible”.The three teachers who coded a negativeresponse did not give enough information toindicate what the reasons were for theirresponse (Table 6).Concluding CommentOverall the papers for level 4 were wellreceived by the teachers and less so by thestudents. Students’ responses suggested that thedifficulty of the items impacted studentsdifferentially with more able students having amore positive experience than less ablestudents. Although, only roughly half of allresponses were favourable and positive, theresults suggest that students found the paperschallenging and interesting. It should beremembered that when asTTle is implementedby teachers, the asTTle software will allowteachers to design suited to the ability level oftheir own students.There were some small areas forimprovement suggested by the participants,including a closer examination of theinstructions for teachers and a box for enteringthe school number. Again it should be notedthat the latter suggestion applies only to thecollection of student data for asTTle normingpurposes and will not be a requirement of theasTTle application when used by teachers.Table 6General comments about papersN %Positive 4 20Negative 3 15Both 0 0No Response 9 45Suggestions 4 20givenTotal 20 100In regards to the three different assessmentpapers, all responses commenting on theseindicated that this was a good way to preventthe students from copying. A typical responsewas: “I liked the 3 different tests, it eliminates3

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!