Outcomes Framework Miller's Framework Description Data Sources ...

OutcomesFrameworkMiller’sFrameworkDescriptionData Sources and MethodsParticipationLEVEL 1Number of learners who participate in theeducational activityAttendance recordsSatisfactionLEVEL 2Degree to which expectations of participantswere met regarding the setting and delivery of theeducational activityQuestionnaires/surveys completed by attendeesafter an educational activityLearning:DeclarativeKnowledgeLEVEL 3AKnowsThe degree to which participants state what theeducational activity intended them to knowObjective: Pre- and post-tests of knowledgeSubjective: Self-report of knowledge gainLearning:ProceduralKnowledgeLEVEL 3BKnows howThe degree to which participants state how to dowhat the educational activity intended them to knowhow to doObjective: Pre- and post-tests of knowledgeSubjective: Self-reported gain in knowledge (e.g.reflective journal)CompetenceLEVEL 4Shows howThe degree to which participants show in aneducational setting how to do what the educationalactivity intended them to be able to doObjective: Observation in educational setting (e.g.online peer assessment and EHR chart stimulatedrecall.)Subjective: Self-report of competence; intention tochangePerformanceLEVEL 5Patient healthLEVEL 6CommunityhealthLEVEL 7DoesThe degree to which participants do what theeducational activity intended them to be able to doin their practicesThe degree to which the health status of patientsimproves due to changes in the practice behaviorof participantsThe degree to which the health status of acommunity of patients changes due to changesin the practice behavior of participantsObjective: Observed performance in clinical setting;patient charts; administrative databasesSubjective: Self-report of performanceObjective: Health status measures recorded inpatient charts or administrative databasesSubjective: Patient self-report of health statusObjective: Epidemiological data and reportsSubjective: Community self-reportMoore DE, Green JS, Gallis HA. Achieving desired results and improved outcomes: Integrating planning and assessmentthroughout learning activities. J Contin Educ Health Prof. 2009;29(1):1-15.

L&D Observational Process ToolObserver ID: ___________ OR ID:__________ Date:______Procedure: (C-section, low risk; C-section, high risk; premature

VCU-OB SA SCALEDirections: Rate your agreement with each item in relation to the simulation you justparticipated in.StronglyDisagreeStronglyAgreeI was confused about where to go when thedecision was made to transport the patient.I knew how to get to the OR once I arrived on the5 th floor.I was confused about my role during the patienttransfer.I was confused about my role during the operativedelivery.I was confused about my role during the transportprocess after the operation.The availability of equipment and resources wassimilar to the L&D OR.1 2 3 4 51 2 3 4 51 2 3 4 51 2 3 4 51 2 3 4 51 2 3 4 53

Using In-Situ Simulation with aFailure Mode Effects Analysisfor Quality ImprovementMoshe Feldman, PhD, MSEllen Brock, MD, MPHCheryl Bodamer PhD, RN, MPHPaul Mazmanian, PhD

Funding and Disclosures• Dr. Mazmanian receives funding from the Centerfor Clinical and Translational Research at VCU.• Collaborators:Moshe Feldman, PhDEllen Brock MD, PhDCheryl Bodamer PhD, RN, MPH6

Context and Setting• Joint Commission standard LD.5.2“Leaders ensure that an ongoing, proactiveprogram for identifying risks to patient safety andreducing medical/health care errors is defined andimplemented.”7

Context and Setting• Identify and mitigate patient safety risksassociated with new surgical patient workflowsdue to a renovation of the L&D unit• Anticipate risks for new and unpredictedworkflows• Evaluate the use of in-situ simulation foridentifying training needs and mitigating patientsafety risks associated with new clinicalworkflows8

Failure Mode Effects Analysis(FMEA)• Interdisciplinary approach to prospectivelyevaluate patient safety risks1. Identify and Map Processes2. Identify Failure Modes3. Estimate & Prioritize Risks9

FMEA: Identify and map processGOAL 1Develop a triageprotocol to minimize riskGOAL 2Develop standardizedwork process for patienttransferGOAL 3Design thesystem tomitigate patientsafety risksGOAL 4Developprocess fortransferringback to L&D1. Room checks atstart of shifts2. Decisionto transport3. Where tooperate?4. Notifyteammembers5. PrepareOR6. TransferPatient toOR7. Performoperativedelivery8.Infantresuscitation9. Transferback toL&D,NICU,10

FMEA: Identify Failure ModesMethods:• Live case observationsWorkflow settingCommunicationHealth informaticsTeam culture• Structured interviews• In-situ simulationsSimulations occurring in the actual clinical setting11

In-Situ Simulation• What does it look like? What can go wrong?L&D 6 th floorStandardized PatientPart task trainerNICU Main 6New WorkflowsSurgery 5 th floorScenarios:‣ Gestational diabetic withprotracted second stage‣ Postpartum hemorrhage‣ Multiple emergencies12

FMEA: Risk AssessmentFailure Modes(what mighthappen)Failure to notify main5 of operativedeliveryCauses(why it happened)Confusion about who isresponsibleEffects Severity Likelihood DetectabilityDelay inoperativedeliveryRiskPriorityScore(LXSXD)• Data sourcesLive observationsStructured interviewsIn-situ simulation13

Risk Assessment: Prioritizing Needs• Risk Priority Number (RPN) = Likelihood X Severity X DetectabilityRiskPriorityLikelihoodSeverityDefinitionThe perceived chance of thefailure happening within a definedperiod.How severe the outcome is to thepatient should failure occur.Detectability Is the area of failure readilyknown, or is it discovered onlywhen a bad outcome occurs?Description of the rating scalesRating of 1-10: from “failure is unlikely” (1 in>5 years) to “very likely or inevitable”(1/day).Rating of 1-10: from “no severity at all”(would not affect individual or system) to“moderate” (significant effect with no injury)to “major injury” to “death.”Rating of 1-10: from “almost certain thecontrol will detect potential cause(s)” to“absolute uncertainty that the control will notdetect potential cause(s) and subsequentfailure mode(s).”Source: Joint Commission Resources 2005, and Failure Modes and Effects Analysis (FMEA). An Advisors Guide, June 2004; Department of DefensePatient Safety Center.14

Planned Change for Patient SafetyFailure Mode Description of Action (Plan) Outcome Measure Person ResponsibleFailure to notify main5 of operativedeliveryRedesign staff role withresponsibility of constantcoordination with main 5 clerkand main 6 charge nurseEvidence ofintervention,observations, staffsurveysPractice committee will redesign staffrolesNursing will post staff responsibilities1. Identify Process6.Implement(Do)2. Identify Failure Modes5. ActionPlan(Plan)Rapid cycleimprovementusingsimulation7.Evaluate(Study)3. Estimate & Prioritize Risks4. IdentifyIntervention(Plan)8. Revise(Act)15

Results• 25 structured interviews with nursing, residents,attendings, L&D teams, front desk clerks• 27 potential failure modes were identified• 11 members of the Obstetrical team ranked failuremodes on likelihood, severity, and detectability• Highest risk priority scores were associated withanesthesia resources, communication between teammembers, and availability of resources• In-situ simulation revealed unique failure modes notdetected with observations and interviews alone16

Evidence of Translation• Implementing the Action PlanFailure ModeCHSPS QI RecommendationOwner/ContactRecommendationsPlan of ActionAnesthesiaunable to staffcases on bothfloorssimultaneouslyon nights andweekendsInability toefficientlycommunicatewith NICU• Increase staffing levels orbackup for anesthesia.• Achieve more predictablepatient throughput throughscheduling.• Develop and disseminateformal workflow roles andprocess. (1,Nursing)• Test the communicationsystem from the Main 5 OR• Mitigating risks• B • A meeting is scheduled on 10/24 withL&D leadership, physicians, Anesthesialeadership, and CRNAs• PerinatalPracticeCommittee• Standard language approved byPerinatal Practice Comm. To be usedwhen calling for NICU resources.Language will include Location, Room#, Gestation, and brief reason for call.• Staff will be education on this duringOrientation• Upgraded Ascom phones“We did our first trial run of a scheduled uncomplicated C/S in the Main 5 OR this morning andwith the exception of a printer problem and a confusion about a phone # … everything wentoff without a problem. There was even a drop in the fetal heart … and the NICU had torespond, but all went well. Not bragging, but great work group!!! Check out the pink footprintsthat now guide the way in the Main OR.”17

Conclusions• In-situ simulation is feasible for identifyingeducational needs and apparently mitigatingrisks associated with new clinical workflows• In-situ simulation allows for rapid training ofinterdisciplinary healthcare teams for newclinical workflows• January 2013 – follow up surveys and focusgroups to assess value of the modified FMEAwith in situ simulation18

Implications for Learners andPlanners• Self assessment and practice based learningWork processesIdentify educational needs• Links CPD to patient safety and systems of care• Tools for assessment, educational planning, andtranslation to change19

Thank you for your attentionQuestions and DiscussionContact usmfeldman@vcu.edu20