What does evidence-based practice in education mean?

Cheri Conference 2006What does evidence-basedpractice in education mean?Dr Kerry HempenstallSenior Lecturer, Division of Psychology,School of Health Sciences,RMIT University, VIC.www.rmit.edu.au/staff/kerry_hempenstall

Evidence-based education(Teachers should) engage with theevidence-based research literature onwhat works for students, with and withoutlearning difficulties, and understand whatconstitutes evidence (National Inquiry into theTeaching of Literacy, 2005).

Why should education embraceresearch?Those fields that have displayedunprecedented development over the lastcentury, such as medicine, technology,transportation, and agriculture have beenthose embracing research as the primedeterminant of practice (Shavelson & Towne,2002).

1990’s Evidence-based medicineStep 1 - AttitudePractitioners learn of the effectiveness andrisks of different interventions, usingreliable measures of benefit and harm as aguide to practice.There is strong support as it providesbetter health outcomes for patients.

Use external sourcesMedicine has Medline(http://medline.cos.com) andCochrane Collaboration(www.cochrane.org)Useful when there is a consensus.

1990’s Evidence-based psychologyAmerican Psychological Associationintroduced empirically supportedtreatment to highlight differentialpsychotherapy effectiveness.

1990’s Evidence-based psychologyThe criteria for a treatment to beconsidered well established included•efficacy through two controlled clinicaloutcomes studies or a large series ofcontrolled single case design studies, and•availability of treatment manuals toensure treatment fidelity, and•provision of clearly specified clientcharacteristics.

So, what might it offer education?Reduce the pendulum swings we’veendured, and redress educationalachievement differences in ourcommunity (Slavin, 2002).

Evidence-based educationIssues:•Acceptance of research as informingpolicy and practice•The role of teacher education•Discerning sound from unsound research

Research can help with many questions•Are reading approaches equally effective?•Is slow initial progress cause for concern?•Can teaching alter the trajectory ofdisadvantaged students?•Is social promotion or retention best?•Do smaller class sizes make a difference?

Evidence-based education:The integration of professional wisdomwith the best available empirical evidencein making decisions about how to deliverinstruction (Whitehurst, 2002).Doesn’t supplant the craft of teaching

Research informing practice?Resistance to using research as guide topractice• Uniqueness vs commonality• Teaching has little effect on outcomes• Research is reductionist

Research informing practice?• Pressure for change is building withineducation.• Its purely artisan cloak challenged• Adopt the demeanour of a researchbasedprofession

Artisan cloak?When experience, personality, intuition,or creativity are the sole determinants ofpractice, we risk inconsistency at theindividual level, and disaster at thesystem level.Consider Galileo: observation vs commonsense

Pressure for reformState and national testing effects:•greater public awareness of studentprogress•government budgetary vigilance•increased attention to the long-termeffects of curricula.

Litigation threat?A Melbourne parent successfully suedBrighton Grammar School for a breach ofthe Trade Practices Act (Aug, 2006).She argued that it had failed to deliver onits promise to address her son's readingproblems.

Erroneous beliefs correctedLearning to read is as natural as learning tospeak (National Council of Teachers of English,1999).Parents reading to children is sufficient toevoke reading (Mem Fox, 2005)Good readers skim over words rather thanattending to detail (Goodman, 1985)

Skilled reading involves prediction fromcontext (Emmitt, 1996)English is too irregular for phonics to behelpful (Smith, 1999)Accuracy is not necessary for effectivereading (Goodman, 1974)

Fluent readers identify words asideograms (Smith, 1973).Children do not learn to read in order tobe able to read a book, they learn to readby reading books (NZ Ministry of Educationdocument, 1988)Good spelling derives simply from theact of writing (Goodman, 1989)

Parent information sheet:X Park Primary School 2003It is inappropriate for your child tobe directed to ‘sound-out’ words,using individual letter sounds, asmany words cannot be identified inthis manner.

XXX PS Teaching your childreading strategies 2002If your child has difficulty with a word:•Ask your child to look for clues in thepictures•Ask your child to read on or reread thepassage and try to fit in a word that makessense.•Ask your child to look at the first letter tohelp guess what the word might be.

Problem Areas:Educational decision-makers havelacked a scientific framework, and beeninclined to accept proposals based ongood intentions and unsupportedopinions (Carnine, 1991)

Problems are internationalU.S. Department of Education reportblamed the “disconnect betweenpractice and evidence” for the lackof progress over the past 30 years inraising student achievement in math,reading and science.

Problem AreasTeachers lack training and direction inseeking out and evaluating researchFewer than one in two hundred teachersare experienced users of the ERICeducational database.(Carnine, 1995)

Problem AreasAustralian Research Council allocated$2.3 million last year on educationprojects, but policy makers and teachersgenerally take little notice(Professor Peter Cuttance, Centre for AppliedEducational Research, 2005).

Problem AreasThe findings of most research reportsrarely reach schools. "The treatment ofresearch by state educationdepartments is based solely on costand what suits the political vision ofthe government of the day."(Ted Brierley, Australian Secondary PrincipalsAssociation president, 2005).

How might reform occur?Research-based educational practices areeither mandated as in Great Britain(Department for Education andEmployment, 1998) or a pre-requisite forfunding as in the 2001 No Child LeftBehind Act (U.S. Department of Education,2002).

Teacher training programsAll members of the teachingprofession should have basicresearch skillsTeacher education programs havenot provided this training. Why not?

Consume and/or conduct research?It is important that, at a minimum,students in training are trained to becomegood consumers of research.In psychology, the principle of scientistpractitioneraccepted.

Need research methodstrainingWhen consensus is yet to be reachedpractitioners should be able to read thesigns

Training in research designIn medicine, psychology, and numerousother disciplines, randomized controlledtrials (RCT) are considered the goldstandard for evaluating an intervention’seffectiveness.In psychology, research methods areincluded in every subject over the degree.

National Inquiry into theTeaching of Literacy (2005)Recommends a national program to:• produce evidence-based guides foreffective teaching practice - thefirst to be on reading;• promote research into the mosteffective means of preparing preserviceteachers to teach reading.

Teacher training programsUnlike many other professions inAustralia, there is no professional bodyaccrediting teacher education at thenational level(Dr Gregor Ramsey, Chair, National AccreditationOf Pre-service Teacher Education, 2005)

National Institute for QualityTeaching and School LeadershipObjective: A strengthened nationalresearch and evidence base to informquality teaching and school leadershipACER is evaluating current teachereducation policies and practices

Asking your own questions•WHO conducted this research?Authoritative, biased?•WHY was the research conducted? Arethe results self-serving? Is thereindependent corroboration?•WHAT is being researched? Does thedata support the conclusions

•WHERE was research done? Similarsetting to yours?•WHEN was the research conducted?•ENOUGH participants for a crediblemeasure?•HOW was the data gathered andanalysed?

Stanovich and Stanovich (2003)1. Findings published in refereed journals.2. Findings replicated by independentresearchers with no stake in theoutcome.3. Consensus in research community aboutreliability and validity of the findings.

Stumbling block:Quality of much educational researchVarying quality of past research producedequivocal or contradictory findings.Clearly a need for higher quality research

National Reading Panel reviewPhonemic Awareness: 1962 studies, 52 metresearch methodology criteriaPhonics: 1,373 studies, 38 met criteriaFluency: 364 studies, 16 met criteriaVocabulary: 20,000 studies, 50 met criteriaComprehension: Of 453 studies, 205 metcriteria

Quality of educational researchJournal of Educational Research (2000-2003), only 6 out of 112 articles usedexperimental/control comparisons.

Finding good advice on program qualityPrestigious researchers peruse the evidence,they•Gather the methodologically soundresearch, and combine the results•Use statistical process, such as metaanalysis,to assemble the results fromthese studies.•Report findings widely.

Meta-analysesanalyses(Lloyd, Forness, & Kavale, 1999)Whole language180 studies 637 effect sizes Overall ES = .09Perceptual/motor training180 studies 117 effect sizes Overall ES = .08Learning styles39 studies 205 effect sizes Overall ES = .14

Meta-analysesanalyses(Lloyd, Forness, & Kavale, 1999, 2005)Early intervention programs74 studies 215 effect sizes Overall ES = .68Direct Instruction (DI)25 studies 100+ effect sizes Overall ES = .82Behavioural treatment of classroom problemsof students with behaviour disorder10 studies 26 effect sizes Overall ES = .93

Age of accountabilityThe fact that a program is based onscientific research does not mean that it iseffective.Programs that have strong, rigorousevidence of effectiveness should beemphasized over those that are only basedon valid principles

Small scale studiesApart from the true experimental studiesthat enable cause to be assigned, wecontinue to need the small scalecorrelational, descriptive, or ethnographicresearch.

Small scale studiesA single study involving a small numberof schools or classes may not beconclusive in itself, but many suchstudies, preferably done by manyresearchers in a variety of locations, canadd confidence that a program's effectsare valid (Slavin, 2003).

Where to find information aboutevidence-based interventionswww.peri.org.au/What Works Clearinghouse: U.S. Department ofEducation - trusted source of scientific evidence ofwhat works in education.Eric Digests Short reports (1,000 - 1,500 words) ontopics of prime current interest in educationPromising Practices Network: Programs forimproving outcomes for children, youth, andfamilies.

International Campbell Collaboration:Systematic reviews of evidence on theeffects of interventions in the social,behavioral, and educational arenas.Social Programs That Work:Programs that are backed by rigorousevidence of effectiveness.Coalition for Evidence-Based Policy.Identifying educational practicessupported by rigorous evidence

Partnership for Reading - a databasewith abstracts of approximately 460research studies related to the teachingof reading in grades K-3.

Comprehensive School Reform ProgramReport on scientifically based research onComprehensive School Reform.Blueprints for Violence Prevention:Initiative for reducing adolescent violentcrime, aggression, delinquency, andsubstance abuse.Florida Center for Reading Research:Information about research-basedpractices related to literacy instructionand assessment for children in P-12

American Institutes for Research (2005):Guide evaluating quality andeffectiveness of 22 primary schoolteaching models.In a major meta-analysis of researchComprehensive School Reform andStudent Achievement, Direct Instructionwas given the highest classification(Center for Research on the Education of StudentsPlaced At Risk, 2002)

Council for Exceptional Childrenmakes judgments on professionalpractices – See what the Alert series saysabout Phonological Awareness, SocialSkills Instruction, Class-wide PeerTutoring, Reading Recovery, MnemonicInstruction, Co-Teaching, FormativeEvaluation, High-Stakes Assessment,Direct Instruction, Cooperative Learning.

www.peri.org.au/Oregon Reading First Centerreviewed and rated 9 comprehensivereading programs. Programs had toinclude materials grades from P-3; andaddress the five essential componentsof reading.http://www.rmit.edu.au/staff/kerry_hempenstall