The Effectiveness of Problem Based Learning 1 - Teaching and ...

TeachingandLearningRESEARCHBRIEFINGOctober 2004 Number 8The effectiveness of Problem Based Learning 1:a pilot systematic review and meta-analysis‘Problem Based Learning’, a cluster of applied approaches to teaching, is increasingly popular inprofessional education both in the UK and internationally. However, there are still many important questionsabout its definition and about the forms of Problem Based Learning which may be best for particularstudents in different contexts. This Research Briefing reports on a pilot systematic review andmeta-analysis, undertaken to evaluate evidence of the effectiveness of Problem Based Learning as definedby the research studies which were cited in five previous reviews.It indicates the challenges of conducting research of this sort and the difficulty of achieving ‘safe’ knowledgeof teaching effectiveness.The term ‘Problem Based Learning’ is usedin relation to a wide range of practices.Descriptions of the Problem Based Learning,and of the control group interventions towhich it was compared, were generallyinadequate to identify the key features of theapproach used.There is a need for consistent reporting of thekey features of the Problem Based Learningapproach used in any particular study.The results of the outcomes reported weremixed. Some effects favoured ProblemBased Learning, whilst others favoured thecontrol group with which Problem Basedlearning was being compared.Previous reviews do not provide sufficientevidence to support the widespread adoptionof Problem Based Learning without furtherresearch. Where Problem Based Learning isintroduced, it should be accompanied byrigorous evaluation.www.tlrp.orgTeaching and Learning Research Programme

The researchBackgroundProblem Based Learning is a majordevelopment in educational practicethat has a large impact across subjectsand disciplines worldwide. It is notalways clear what exactly has beendone in the name of Problem BasedLearning. There is also a growingnumber of references in the literature toadapted or “Hybrid” Problem BasedLearning courses and to coursesdescribed as 'Enquiry' or 'Inquiry'Based learning. Problem BasedLearning has arguably been one of themost scrutinised innovations inprofessional education. However,despite the volume of literature onProblem Based Learning it is not at allclear that we have safe knowledgeabout the effects of Problem BasedLearning in different learning contextsand in different modes of operation.Whilst the principle of research reviewsis well established in education, theappropriate process and purpose forthem is not. Systematic reviews can bea valid and reliable means of avoidingthe bias that comes from the fact thatsingle studies are specific to a time,sample and contexts and may be ofquestionable methodological quality.They attempt to discover theconsistencies and account for thevariability in similar appearing studies.There is a consensus emerging aboutthe need for systematic reviewscovering selected topics in education.Such reviews will identify the existingevidence, provide at least someanswers to the review questions andprovide directions for future primaryresearch.Objectives of thepilot review• To establish the level and quality ofexisting ‘safe’ knowledge about theeffectiveness of Problem BasedLearning based on previouslypublished reviews• To confirm the need for a fullsystematic review of theeffectiveness of Problem BasedLearning• To establish the value of the methodof systematic review used• To identify and clarify any problemswith the review protocol, processand instrumentsReview questionsThe initial review questions were…When compared to other approachesdoes Problem Based Learning result inincreased participant performance at:• adapting to and participating inchange;• dealing with problems and makingreasoned decisions in unfamiliarsituations;• reasoning critically and creatively;• adopting a more universal or holisticapproach;• practising empathy, appreciating theother person’s point of view;• collaborating productively in groupsor teams;• identifying own strengths andweaknesses and undertakingappropriate remediation (self-directedlearning)?MethodsReview DesignThe review followed the approach usedby The Cochrane Collaboration, aninternational effort to identify evidenceof effectiveness in the field of healthcare. The design of the review protocol,data extraction tools and overall reviewprocess is based on the guidance forreviewers produced by The CochraneEffective Practice and Organisation ofCare Review Group and the NHSCentre for Reviews and Dissemination.This approach has been developedfrom the emerging ‘science’ ofsystematic reviewing in order minimisebias and errors. The review protocolwas registered with the CampbellCollaboration. Copies of the reviewprotocol and instrumentation areincluded in the full project report.Inclusion criteria• The review included participants inpost-school education programmes.• Study designs included wereRandomised Controlled Trials (RCT),Controlled Clinical Trials (CCT) ,Interrupted Time Series (ITS), andControlled Before & After studies(CBA). Qualitative data collectedwithin such studies e.g. researchers’observations of events, wereincorporated in reporting. Studiesthat use solely qualitative approacheswere not included in the review.• The minimum methodologicalinclusion criteria across all studydesigns were the objectivemeasurement of studentperformance/behaviour or otheroutcome(s) and relevant andinterpretable data presented orobtainable.Minimum inclusion criteria forinterventions (i.e. what is ProblemBased Learning) were set out in thereview protocol. However, reporting wasinsufficient to establish whether thesecriteria were met in virtually all theprimary studies considered.Review ProcessThe review coordinator examined thereviews and identified the relevantcitations. For each citation either anabstract or full text copy was obtained.Two members of the review team thenscreened the citations to eliminatethose that obviously did not meet theminimum inclusion criteria. Theincluded papers were then distributedamongst the reviewers for qualityappraisal and data extraction. Eachpaper was reviewed independently bytwo reviewers. Where there weredifferences of opinion betweenreviewers the review coordinator alsoreviewed the paper. At all stages, theprocess used and outcomes areexplicit. A full list of included andexcluded studies is provided in the fullreport. The completed qualityassessment and data extraction toolswere returned to the review co-ordinatorwho lead the process of producing areport of the review synthesising andanalysing the results where appropriate.Results91 citations were identified from the fivereviews. Of these 15 were judged tomeet the inclusion criteria. Of the 15only 12 reported extractable data. Thediscussion below refers only to these 12studies. The studies all reported onProblem Based Learning used in highereducation programmes for healthwww.tlrp.orgTeaching and Learning Research Programme

professional education at both pre andpost registration levels. The majority ofstudents were in medicine and themajority of these studies reported onpre-registration medication education.The majority of reported effects were forwhat appeared to be tests of‘knowledge’ using multiple choiceformats. There were also a smallnumber of effects reported on 'impacton practice' and on 'approaches tolearning'. Very little information wasgiven in the papers from which datawas extracted about the design,preparation or delivery processes ofeither the Problem Based Learningintervention or the control to whichProblem Based Learning was beingcompared. Four of the studies used arandomised experimental design, two aquasi-randomised experimental designand the remainder were controlledbefore and after studies. Only onestudy reported standard deviated effectsizes. Effect sizes were calculated forall the other extracted effects with theexception of three studies whereinsufficient data were provided in thestudy report.The studies that reported 'effects onpractice' all used different outcomesand measurement instruments. In onlyone of the studies that reported 'effectson practice' was sufficient data providedto calculate effect sizes. This makes itdifficult to synthesise the study results.One study reported 'attitudes topractice' and found effect sizes thatfavoured Problem Based Learning.Another measured 'nursing processskills' and of the seven effects reportedfive favoured the control group. Thethird study reported 'consultation skills'and on all the effects reported theresults favoured the control group.However, in this particular study, whichused a quasi – experimental design, thenature of the Control group interventionand the outcome measures used wouldappear to have put the control group ata distinct advantage.Two studies reported effects on'approaches to learning'. The twostudies used different instruments andreported five effects. In both studies,the results favoured Problem BasedLearning on all the scales. The ProblemBased Learning groups had fewer of theundesirable and more of the desirable'approaches to learning' after theintervention. However, the overallMajor implicationsThe review found comparatively fewstudies cited in the five sample reviewpapers whose design and methodologywere good enough to eliminate bias andprovide precise estimates of effect.Studies that achieved this wereconcentrated largely in medicaleducation in North America and Europeand focussed mainly on measuring'accumulation of knowledge' usingmultiple choice assessments.However, an outcome of 'accumulationof knowledge' and in particular itsassessment through multiple choiceinstruments is not regarded by manyadvocates of Problem Based Learningas congruent with the approach itself oras a valid indicator of success. Thesecond issue is how generalisable theseresults are to other disciplines and toother cultural contexts.Figure 1: Effect size for reported study outcomes focussing on 'knowledge'picture was of deterioration in theapproaches to learning of both theProblem Based Learning and controlgroups, which Problem Based Learningappears to have mitigated.Figure 1 illustrates the effect sizes forreported outcomes that could usefullybe grouped under a heading‘knowledge’. Effect sizes ranged from d=-4.9 to 2.0. There were sufficienteffects reported to justify an attemptedmeta analysis. The meta analysis of theoutcome ‘knowledge’ included 14effects reported in 8 different studies.Concerning the review questions, theresults of the meta analysis are largelyinconclusive. The mean effect size wasd= -0.3 but the 95% confidence intervaldid not exclude an effect size of d = 0.Sensitivity analysis suggested that studydesign, randomisation, level ofeducation and assessment format areall potential moderating variables but theresults were not conclusive. Importantlythe 95% confidence intervals do notexclude potentially ‘large effect sizes ofd = + 1.0 or – 1.0. An effect size of d= 1.0 would mean that 84% of studentsin the control group were below thelevel of the average person in theProblem Based Learning group. Aneffect of this size would appear to haveimportant practical significance.Only one study reported effects on'satisfaction with the learningenvironment' that met the reviewinclusion criteria. The study in anundergraduate medical educationprogramme required students to ratetheir experience on a series of scales.On all except two of the nine effectsreported, the effect size favoured theProblem Based Learning group. Thelargest effect size in favour of ProblemBased Learning was the students ratingof innovation (d=1.6). The largest effectsize in favour of the Control group wasstudents’ rating of clarity (d= -1.0).Teaching and Learning Research Programmewww.tlrp.org

FurtherinformationFurther information about the project canbe downloaded from the project website(address below). A detailed summary ofthe two empirical studies can bedownloaded from the ESRC Regardwebsite (www.regard.ac.uk). A fullreport of the Pilot Systematic Reviewand Meta-analysis was published by theLearning & Teaching Subject NetworkCentre for Medicine, Dentistry andVeterinary Medicine and can bedownloaded from its website www.ltsn-01.ac.uk. A full report on the evaluationof Problem Based Learning inContinuing Nursing Education isavailable from the project website.Systematic Review teamPiet Van den Bossche (University ofMaastricht, The Netherlands);Charles Engel (Centre for HigherEducation Studies, Institute ofEducation, University of London);David Gijbels (University of Antwerp,Belgium); Jean McKendree (Learningand Teaching Support Network forMedicine, Dentistry and VeterinaryMedicine, University of Newcastle);Mark Newman (Middlesex University);Anthony Roberts (South Tees HospitalTrust, North Tees Primary Care Trust andUniversity of Durham); Isobel Rolfe(Faculty of Health, University ofNewcastle, Australia); John Smucny(State University of New York UpstateMedical University, USA);Giovanni De Virgilio (Istituto Superioredi Sanità, Italy).The warrantThe research question in this study wasspecifically concerned with establishingwhether the use of activities termed ProblemBased Learning results in different‘outcomes’ to the use of other teaching andlearning approaches. The review used asystematic and transparent process that hasbeen demonstrated to provide a more validand reliable appraisal of existing researchevidence that allows for the development andaccumulation of ‘knowledge’ specifically by:• Defining transparent inclusion criteriathat limit consideration to studies thatused research designs which have beenempirically established as optimal forproviding evidence about effectiveness• Using these criteria to make decisionsabout the inclusion of studies, theresults of studies and the interpretationof these results• Two reviewers independently appraisedthe quality of these studies and carriedout data extraction• The use of existing frameworks for thereporting of Problem Based Learning asa basis for providing descriptivesummaries of the included studies• The use of an explicit framework for theanalysis of the results from the individualstudies using both systematic narrativesynthesis and exploratory meta-analysisThe limitations of the study design and itsconduct and their consequences have beenreported and explored. The relationshipbetween the results and conclusions istherefore explicit and the limited conclusionsdrawn do not go beyond that justified by theresults of the study.Teachingand LearningResearch ProgrammeTLRP is the largest education researchprogramme in the UK, and benefits from researchteams and funding contributions from England,Northern Ireland, Scotland and Wales. Projectsbegan in 2000 and will continue withdissemination and impact work extendingthrough 2008/9.Learning: TLRP’s overarching aim is toimprove outcomes for learners of all ages inteaching and learning contexts within the UK.Outcomes: TLRP studies a broad range of learningoutcomes. These include both the acquisition of skill,understanding, knowledge and qualifications and thedevelopment of attitudes, values and identities relevantto a learning society.Lifecourse: TLRP supports research projects and relatedactivities at many ages and stages in education, trainingand lifelong learning.Enrichment: TLRP commits to user engagement at allstages of research. The Programme promotes researchacross disciplines, methodologies and sectors, andsupports various forms of national and international cooperationand comparison.Expertise: TLRP works to enhance capacity for allforms of research on teaching and learning, and forresearch-informed policy and practice.Improvement: TLRP develops the knowledge base onteaching and learning and collaborates with users totransform this into effective policy and practice in the UK.TLRP is managed by the Economic and SocialResearch Council research mission is to advanceknowledge and to promote its use to enhance thequality of life, develop policy and practice andstrengthen economic competitiveness. ESRC isguided by principles of quality, relevance andindependence.PEPBL website:http://www.hebes.mdx.ac.uk/teaching/Research/PEPBL/index.htmPEPBL Grant holder and Principal Investigator: Mark NewmanPEPBL contact: Mark NewmanE-mail: m.newman@ioe.ac.uk Tel: +44 (0)20 7612 6575Social Science Research UnitInstitute of EducationUniversity of London, 18 Woburn Square London WC1H 0NRTLRP Directors’ Team❚Professor Andrew Pollard ❚ Cambridge❚Dr Mary James ❚ Cambridge❚Dr Kathryn Ecclestone ❚ Newcastle❚Dr Alan Brown ❚ Warwick❚John Siraj-Blatchford ❚ CambridgeTLRP Programme Office❚Dr Lynne Blanchfield ❚ Lsb32@cam.ac.uk❚Suzanne Fletcher ❚ sf207@cam.ac.ukTLRPUniversity of CambridgeFaculty of EducationShaftesbury RoadCambridge CB2 2BX UKTel: +44 (0)1223 369631Fax: +44 (0)1223 324421www.tlrp.org