Towards evidence-based practice in science education 1

TeachingandLearningRESEARCHBRIEFINGNumber 1Towards Evidence-based Practicein Science Education 1Using diagnostic assessment to enhance learningMuch research has been carried out on students’ understanding of key science ideas, butthis has not led to marked improvement in teaching and learning. As a means of improvingpractice, banks of diagnostic questions, based on research, were developed for several corescience topics. These were used to monitor students’ understanding of key science ideas,and to explore how the provision of research-based materials of this sort can influenceteachers’ practices and students’ learning.• Science teachers’ practice, and students’learning, can be significantly enhanced byproviding teaching materials that embodyresearch findings and insights.If findings and insights from research are‘translated’ into specific practical implications,or teaching materials, the likelihood and scaleof their impact on practice is greatly increased.• Carefully designed probes, based onresearch, can provide quality informationon students’ understanding of key scienceideas, and inform subsequent action.Tools for quickly ‘measuring’ understanding ofkey ideas can help focus learning activity andindicate levels of understanding across aclass. More should be developed and madeavailable to teachers.• The level of students’ understanding ofmany fundamental science ideas is low,and increases only slowly with age.Current teaching approaches in science donot result in widespread understanding ofmany core ideas. Levels of understanding ofa few key ideas should be monitoredsystematically over time, to inform curriculumdecisions.www.tlrp.orgTeaching and Learning Research Programme

The researchThe EPSE NetworkThis project is one of four undertaken by theEvidence-based Practice in ScienceEducation (EPSE) Research Network. TheNetwork is a collaboration involving theUniversities of York, Leeds, Southampton andKing’s College London. Its overall aim is toexplore ways of enhancing the impact ofresearch on practice and policy in scienceeducation, by improving our understanding ofthe interface between researchers andpractitioners. The EPSE Network hasdeveloped and evaluated several examples ofevidence-informed practice, and has exploredpractitioners’ perceptions of the influence ofresearch on their practice. Whilst focussingon science education, the findings andoutcomes may also illuminate the researchpracticeinterface in other subject areas.Background andrationaleOver the past 30 years, a great deal ofresearch has been carried out in manycountries on learners’ ideas about the naturalworld. This has helped identify commonlyheldideas which differ from the acceptedscientific view, and has shown that these areoften very resistant to change. Thesefindings have clear implications for the paceand sequence of instruction in many sciencetopics, particularly those which involveunderstanding of fundamental ideas andmodels. Yet whilst many teachers know ofthis research, it has had little systematicimpact on classroom practices, or onscience education policy in the UK.One response might be to try tocommunicate more effectively to teachers thefindings of research on science learning.This, however, seems rarely to be an effectivestrategy for changing educational practices.Instead, this project sought to influenceteachers’ practice by providing them withtools derived from research – a bank ofdiagnostic questions, based on the kinds ofprobes used by researchers. This wouldenable them more easily to collect data ontheir own students’ learning of key ideas, andso adjust their teaching in the light of thisevidence of their own students’ learning.There is also another sense in which thisapproach can be seen as an example ofevidence-based practice. A review by Blackand Wiliam (1998) of research on the use offormative assessment showed that this canlead to significant learning gains, throughclarifying learning objectives, and providinginformative feedback to learners. A possibleobstacle, however, is the difficulty ofgenerating good formative assessmentquestions and tasks. By providing teacherswith a large bank of diagnostic questions, wecan explore the extent to which thisfacilitates change in practice, leading tobetter learning.www.tlrp.orgWhat we didThe first stage of the project was to developbanks of diagnostic questions. This wasdone in collaboration with a ‘partnershipgroup’ of practitioners, which includedseveral primary and secondary teachers, LEAadvisers, and writers of teaching materials.This group chose the science topics for thiswork: electric circuits, forces and motion,particle models of matter, and life processes(digestion, respiration and photosynthesis),and acted as a critical ‘expert’ review groupthroughout. Diagnostic questions weretrialled in the partner teachers’ schools, withfollow-up interviews with some students toprobe their understanding further.Questions with a two-tier structure (Figure 1)were found to be particularly useful. The first‘tier’ asks for a prediction about what will beobserved – and the second then asks for thebest explanation for this. A correct answercombination is a good indicator ofunderstanding. Other answer patternsindicate common misconceptions.Questions were also developed with moreopen response formats, and to stimulatesmall-group discussion.The banks of diagnostic questions producedwere then used in two ways:• to evaluate students’ currentunderstanding of some key ideas at theends of Key Stages 2, 3 and 4;• to monitor how a panel of teachers useddiagnostic question banks in their ownteaching, and the impact on classroomactivity and student learning.Understanding ofkey ideasRather than assess understanding across thesciences, as other surveys have done, wechose to probe students’ understanding of afew ideas in depth. The samples of studentstested came from a range of schools, andwere above the national average for eachage group (from predicted national testscores). Two important outcomes emerged.First, the proportion of students able todemonstrate understanding of some verybasic ideas in these topic areas was low, andoften increased little with age. Fewer than50% at age 14 understood that electriccurrent is the same everywhere around aseries circuit; the proportion at age 16 wasalmost identical. Ideas that build on this,Q2current current other TOTALthe same graduallyused upQ1 current the same 65 46 4 115current gradually used up 11 63 4 78other 6 7 4 17TOTAL 82 116 12 210Table 1 Models used by Key Stage 3 pupils in two questions on electric currentabout the relationships between voltage,current and resistance in simple circuits, weregrasped by fewer than 20% of 16 year-olds.Less than a quarter showed soundunderstanding of the forces acting on objectsmoving at steady speed or slowing down.Only about half at age 14 (rising to about60% by age 16) understood that mass isconserved in physical and chemical changes.Over 50% at age 14 wrongly classified someparticle-level illustrations of mixtures andcompounds, and of physical and chemicalchange. These results are not especially newor surprising – indeed they corroboratefindings of studies that predate the NationalCurriculum. But they indicate that currentapproaches to the teaching of science do notlead, for many students, to a soundunderstanding of many basic explanatoryideas – and that this is not highlighted bycurrent measures of student performance.A second important outcome concerns howwe measure ‘understanding’. Most researchstudies, like examinations, probe each keyidea with a single question. Instead, weused several, to explore the consistency ofindividual students’ responses. We foundsurprising variation in the models used indifferent questions probing the same idea.Table 1 shows one example. Of the 115students who used the correct scientificmodel in Q1 (in Figure 1), only 65 did soagain in Q2. Either question alone is adubious indicator of understanding. Wefound similar inconsistencies in answers onother science ideas. Clearly any measure of‘understanding’ must include severalquestions on each idea tested, and state acriterion for ‘understanding’. Agreedoutcome measures are a pre-requisite forexperimental comparisons of programmes orteaching approaches. Even for topics wheretargets can be readily agreed, developingsuch measures is a considerable task.Helping teachersuse diagnosticassessmentIn the final phase of the project, we exploredhow provision of diagnostic materials mightinfluence teachers’ practices and studentlearning. Twenty teachers, including somefrom the project partnership group, weregiven a bank of diagnostic questions for achosen science topic, with some suggestionson possible ways of using them. Interviewswith these teachers explored how they usedthe materials and its impact on their teaching.Teaching and Learning Research Programme

Not surprisingly, some teachers’ initialreaction was ‘oh good, questions – we’re soshort of useful questions’. Several chosefrom the banks to make up new end-of-topictests: ‘we needed a new test … so I lookedin the pack … and just pulled a selectionout’. A few talked of using the banks todevelop pre-tests for a new topic, ‘to find outwhat their initial ideas were before we thenmoved on’. This opens up the possibility ofmore responsive teaching, taking students’existing ideas into account.Many teachers were hesitant about increasingthe amount of testing: ‘we do so many tests.I wanted a more flexible way, that wouldenable me to see where the problems were intheir thinking and move them forward.’ Othercomments strongly reinforced this viewpoint.One teacher commented that she had initiallyseen the banks as test items but quicklycame to the view that: ‘the use for me isopening up discussion, thinking about howthey’re actually perceiving things.’ Strikingly,more than half of the group highlighted:‘discussion with the pupils … these haveprompted a great deal, which wouldn’t havebeen there without them’. For some, thiswas a development of their teaching style.One acknowledged that, ‘if I was the sort ofteacher that was always promptingdiscussion, then it probably wouldn’t havebeen a necessity, I wouldn’t have neededthat. But I did and it’s helped, withoutquestion. I’m having more discussions inclass than previously.’ For another it was amatter of seeing how to use: ‘a much moreinteractive, discursive approach … a style ofteaching I prefer. … perhaps I haven’t beenconfident in physics before to risk it. This hasgiven me an impetus.’ An NQT felt that thequestion banks helped clarify her ownunderstanding, and highlighted key points toteach.Almost all were positive about students’engagement in lessons using diagnosticquestions, with several commenting thatmultiple-choice questions prompted morelively discussion than open-response ones,as they obliged students to evaluatealternative viewpoints: ‘What I got, from oneEPSE question, was an entire unplannedlesson, with pupils fully engaged and makingreal progress with their thinking.’The majority of teachers involved in the studyreported changes, of varying degree, in theirteaching styles, or their approach to thescience topic in question. All planned tocontinue using the materials and teachingapproaches they had begun to explore. Allwere convinced that students’ understanding– and in some cases also their ownunderstanding – of the ideas that reallymatter had been significantly improved. Thisexploratory study demonstrates the potentialfor influencing practice through the provisionof teaching materials based on research, anddeveloped through collaboration betweenteachers and researchers.ReferenceBlack, P. & Wiliam, D. (1998). Inside the BlackBox. Raising standards through classroomassessment. London: King’s College,School of Education.Major implicationsThis work has shown that teachers’ practicecan be significantly influenced by makingavailable teaching materials based onresearch findings and insights. In thisproject, the practices of a group of teacherswere significantly enhanced (in their own viewand that of the researchers) by access tobanks of diagnostic questions informed bythe findings of research on concept learning,and by researchers’ analyses of subjectmatter and experience in probingunderstanding. These helped teachers toidentify more precisely, and to focus teachingmore strongly on, the key ideas that are atthe heart of an understanding of thesescience topics, and which provide a basis forfurther learning. Teachers valued how thequestions enabled them quickly to assessthe understanding of all the students in aclass, rather than sampling a few individuals.They also found structured diagnosticquestions particularly useful for stimulatingsmall-group and whole-class discussion, andreported high levels of student engagement,and lively debate about ideas andexplanations, often providing clear evidenceof student learning. Several felt thatdiagnostic materials helped them to teachscience topics outside their specialist area inmore interactive ways, and with a clearerunderstanding of which ideas to emphasise.Similar banks of diagnostic materials areneeded for other science topics with similarconceptual demands, perhaps drawing onthe experience of current national projects inNew Zealand and Australia to develop webbasedbanks of assessment resources forteachers. Ways of encouraging moresystematic formative assessment using thesematerials also need to be further explored,perhaps by restructuring our materials or byproviding more targeted CPD. Linking our‘provision of materials’ approach to othercurrent projects seeking to develop formativeclassroom assessment might be a means ofincreasing impact on practice.The large banks of diagnostic questionsproduced are in themselves a major outcomeof this project. They embody a huge amountof accumulated knowledge from research,Their ‘translation’ from research tools intoteaching materials involved significantcreative input; the outcomes are newartefacts. Articulation of findings in the formof specific practical implications or usabletools needs to be more generally and widelyseen as a necessary and critical part of theprocess by which educational research mightinfluence practice. The development ofdiagnostic question banks required detailedanalyses of subject content in each sciencetopic; learning objectives then had to beoperationalised as tasks providing differentialdiagnosis (of students who do/do notunderstand a point). This is a severe test ofthe way ideas are stated and sequenced incurricula and teaching schemes. In ourwork, it highlighted significant problems inthe sequence of ideas in several areas,particularly mechanics (forces and motion).Here key ideas in the learning sequence (likethat all forces arise from interactions and soalways come in pairs) are unduly delayed,and some critical ideas (like identifying clearlywhich object each force is acting on) areinsufficiently emphasised. By drawinginfelicities of this sort to attention, and byclarifying expectations of what studentsshould be able to do, the development ofdiagnostic questions covering the wholescience curriculum could lead to significantimprovement of practice and outcome.Finally, this work has shown that the level ofunderstanding among students of many keyideas in science is low, that in many cases itincreases only very slowly with age, and thatwell-known misconceptions are prevalent. Itseems clear that current approaches to theteaching of science do not developwidespread understanding of many coreideas in the science domains studied. This,however, is not inevitable. Teachers, whenusing our diagnostic question banks, notedthat many students actually engage muchmore strongly with these basic ideas, if giventime and space to think about them, thanwith their normal science diet. It also seemsclear that current measures of studentperformance in science do not detect andhighlight these gaps in fundamentalunderstanding. Whilst it is not possible tocollect exhaustive data on every idea thatmight be deemed important, somesystematic data collection should beundertaken, and maintained over time, tomonitor levels of understanding of a few keyideas as students progress through theirschool careers, as a means of monitoring thecurriculum and its effectiveness.Figure 1 A two-tier question on electriccircuitsTeaching and Learning Research Programmewww.tlrp.org