here - EENet

IMPROVING ASSESSMENT AND PERFORMANCEMONITORING IN ONTARIO’S ADDICTION SYSTEMFall 2012“There’s a lot of really good work that’s taken place—anda lot of people involved.”—Dr. Brian Rush, Project LeadA suite of projects funded by the national Drug Treatment Funding Program (DTFP),which aims to strengthen substance abuse treatment systems, is making wavesacross Ontario. Many of these projects have benefitted from the active participationof addictions and mental health agencies—and the agencies themselves haveexperienced changes in their daily practice. Four of the projects, being led out of theCentre for Addiction and Mental Health, are linked: Screening, Assessment, and Recovery Monitoring (SARM) Project Costing Project (also known as the Assessment/Benchmark AddictionsTreatment Costs Project) Client Perceptions of Care Project (also known as the Client SatisfactionProject) DATIS Project (also known as the Addiction Treatment Data Elements andProvincial Service Categories Project)Clockwisefrom top left:April Furlong,Dr. NooshinRotondi, Dr.Yanka Cvetanova,andEmily Hansson.In January 2012, agency staff from each of the DTFP pilot sites, completed four days of training in Illinois towards achievingGAIN‐I Local Trainer certification. Training participants listed from left to right: Karen James, Alison Stagg , Kerry Maloney,Sandy Watson‐Moyles, Lori Griffith, Dominique Morisano, Linda Sibley, Pamela Hill, Mary Boyce, Rob Wabegijig, Lisa Boulay,Marisa Tacconelli Termine, Sonya Lynne Trotter, Jeanne Burns , Lise Bazinet, April FurlongThis Fall 2012 update provides an overview of the recent activities of these four projects,which have seen substantial momentum over the past few months.GETTING THEWORD OUTTraditionally, researchfindings have beenreleased at the end of astudy, but we’vealready been gettingthe word out! Thesefour projects have beenkeeping stakeholders inthe loop and collectingfeedback through allphases of the work.For example, the DATISProject and the ClientPerceptions of CareProject recentlypresented their workto‐datein webinarshosted by EvidenceExchange Network(EENet).This integratedapproach to knowledgeexchange has alreadyhelped generate someinterest in the toolbeing piloted by theClient Perceptions ofCare Project. You canwatch the webinarshere: eenet.ca/products‐tools, underthe heading, “Meetingof the Minds.”

COSTING PROJECT TAKES ON COSTLY ISSUESAssessing and comparing the costs of addictions treatment services playsa key role in measuring the performance of individual treatment services.Standardized cost measures can provide comparative benchmarks. Theycan also inform strategies to improve the efficiency and overall costeffectivenessof the treatment system. The Assessment/BenchmarkAddictions Treatment Costs Project – or Costing Project – is working todevelop and pilot a standardized program of cost reporting andcomparisons for addictions treatment services in Ontario.Following a review of definitions andfunctional centres for addiction services,and after extensive consultations withstakeholders, the project team identified awide range of issues with the potential toimpact the outcome of the project. Inconsultation with eight addiction centrepilot sites, the project team developedguidelines for testing how best to addressthese issues.“It’s not at all clearwhat services cost inthe province. It’s ahuge vulnerability inthe system.”—Dr. Brian Rush, ProjectLeadSince the spring, the eight sites have been providing data and feedback.Based on the results of the pilot testing, which will end March 31, 2013, theproject team anticipates that it will have to carry out a number of activitiesincluding making recommendations to improve the reliability and validity ofreporting financial statistical data, and developing guidelines to estimateand interpret cost ratios for substance use treatment centres.Dr. Brian Rush, Group Head of theHealth Systems and Health EquityResearch Unit, CAMH and DTFPProject LeadADVISORY COMMITTEE WEIGHS INIt was raining outside, but the energy inside the room was notdampened. On September 14, 2012, the Advisory Committee (see pg.5 for list of members) for these four projects met in Toronto. Themeeting brought together the perspectives of diverse stakeholders,including the Ministry of Health and Long‐Term Care, Local HealthIntegration Networks (LHINs), service providers, researchers, andothers. The Committee asked questions, made comments, andoffered useful feedback. Participating pilot agencies have foundvalue in these projects and the tools being piloted. A central theme,therefore, was around whether participating agencies will be able tocontinue to use the new processes and tools. The event was led byDr. Brian Rush, the Lead on three of these projects—but it was atruly collaborative day. As Dr. Rush noted, “The dedication of theteam and the agencies needs to be acknowledged.”Clockwise from top left: April Furlong, Dr. Nooshin Rotondi, Dr. Yanka Cvetanova, and Emily Hansson.2

THE (preliminary) RESULTS ARE IN!Measuring client satisfaction provides an important perspective on the quality and performance ofmental health and substance use treatment services and systems. But while many treatmentprograms in Ontario currently assess client satisfaction, it is not being done consistently. The ClientPerceptions of Care Project is reviewing, developing, and piloting an evidence‐based, sustainable,and standardized tool to help agencies get feedback on the care they provide from those who accessmental health and addictions services.Between April 1 and June 30, 2012, 23 publicly‐funded mental health and addiction treatment agenciesacross the province piloted the Ontario Perceptions of Care – Mental Health and Addictions (OPOC‐MHA) tool. The agencies represented a diverse range of clients and services, and included 82different programs in all. A total of 1758 questionnaires were completed—most (83%) by registeredclients with mental health and/or substance use problems, and smaller percentages by family, friends,and other supports. (To validate the tool against a recognized standard, the Client SatisfactionQuestionnaire (CSQ) was also completed in most agencies).Feedback about the OPOC‐MHA was also collected from interviews and surveys of agency staff. Theresults will inform analyses of the newly developed tool. It is anticipated that some revisions to theOPOC‐MHA will be required—for example, wording changes to make the tool more accessible toyouth. Preliminary results, however, are very positive, and there is already substantial interest in thetool from the field.TELLING THE DTFP STORYEvaluating the impact of a project is an integral part ofunderstanding how system change happens. In itsevaluation of the DTFP Ontario Systems Projects, theEvaluation Centre for Complex Health Interventions isincluding two of the projects outlined above as casestudies: the recovery monitoring component of theSARM Project, and the DATIS Project. The EvaluationCentre’s goal is to understand how multiple projectscan achieve system change. It aims to capture thesystem‐level changes that are necessary to bring aboutan improvement and how the individual projects willcontribute to these changes. The evaluation team isconsidering the perspectives of multiple stakeholders.In the end, the evaluation team will develop a detailednarrative for each case study, describing the process ofDr. Sanjeev Sridharan, Director of the Evaluation Centre for ComplexHealth Interventionsproject implementation, alignment with the other DTFP Ontario Systems Projects, and progresstowards outcomes. The case studies have been chosen in part to reflect the three DTFP investmentareas: Implementation of Evidence‐informed Practice; Strengthening Evaluation and PerformanceMeasurement; and Linkage and Exchange.To date, the team has completed interviews with project leads and staff, and has conducted a shortsurvey with advisory committee members. Future work will include interviews with a sample ofadvisory committee members, and site visits to agencies piloting the SARM tools and procedures.3

Outside Manitoulin Community Withdrawal Management Services, a DTFP pilot agency. Top (l to r): Barbara Kosky, Barb A. Deschamps, Rob D.Wabegijig, Kristie Fabian, Bottom (l to r): Nancy Chau, Dr. Nooshin Rotondi, Bridget DouglasGAINing GROUND IN ONTARIOScreening and assessment tools help service providers determine the appropriate level and intensity ofcare for clients who are entering the substance use treatment system. The Screening, Assessment, andRecovery Monitoring (SARM) Project is drawing on the best available evidence as well as the feedbackof key stakeholders throughout the province to develop and pilot new screening and assessment toolsand procedures. The SARM Project is also piloting an evidence‐based protocol for recovery monitoringsince monitoring the outcome of clients is a crucial part of the process of measuring the performance ofthe system. The project team hopes to determine if it is feasibleto implement a recovery monitoring system for Ontario.For three months, the participating agencies have been pilotingthe SARM protocol in four agencies across the province. (A fifthsite will start recruiting clients in fall 2012.) As of the end ofSeptember, 212 clients have joined the study! These clientsreceive the standard assessment protocol: the Admission andDischarge Criteria and Assessment Tools (ADAT), which iscurrently in place for publicly funded Ontario substance useagencies. Clients also receive the newly developed screening and“We really want toacknowledge how muchthe sites have done forthe project. They’vebeen amazing.”—Dr. Nooshin Rotondi, SARMProject teamassessment protocol, and are monitored by agency staff during the course of their treatment. At severalpoints, clients participate in a follow‐up interview. Study recruitment extends through September of thisyear (November for the fifth pilot site). All interviews will wrap up by the end of February 2013.While the formal results from the SARM Project are not yet in, preliminary feedback from the pilot siteshas been positive, particularly related to the GAIN‐Q3 instrument. This widely used tool was selected forinclusion in the project based on its comprehensive assessment of several life areas, including mentalhealth, its high validity and reliability, its administration time, its relatively low cost, and its ability toautomatically generate treatment planning reports. The Ontario version of the tool reflects substantialinput from key stakeholders in the province’s addiction treatment system. (continued on next page.)4

(continued from previous page)Preliminary feedback from the pilot sites also suggests that the GAIN‐Q3 is more user‐friendly whenfacilitated by the GAIN Assessment Building System (ABS), a web‐based application. The SARM project hasalso resulted in substantial capacity building around the GAIN instruments. To date, of the 14 pilot agencystaff who attended training, 12 have achieved Administration Certification. Ten have also achieved LocalTrainer Certification, an accomplishment that represents many, many hours of work—all done on avoluntary basis! These Local Trainers can now train staff at their own agencies and recommend them forAdministration Certification.Ultimately, by using a common, evidence‐based approach, the SARM Project aims to improve the quality ofthe screening, assessment, and outcome monitoring processes in Ontario’s substance use treatmentsystem. A more effective system will benefit funders, administrators, researchers, clinicians, and clients.DTFP ADVISORY COMMITTEEAnita Webb, Canadian Mental Health Association, KenoraBranchChristine Bois, Formely with Centre for Addiction and MentalHealth (CAMH)Anne Bowlby, Ministry of Health and Long‐Term CareBeata Wezyk, Health Data Branch, Ministry of Health and Long‐Term CareBeth Powell, CAMHBrad Davey, ConnexOntarioBrian Rush, Health Systems and Health Equity Research Groupand Provincial Systems Support Program, CAMHCate Sutherland, Addiction CentreClaudio Rocca, Drug and Alcohol Treatment Information System(DATIS)Cynthia Damba, Toronto Central Local Health IntegrationNetworkDan Purdon, G & B HouseDanielle Layman‐Pleet, Ministry of Health and Long‐term CareDavid Ross, National Centre for Operational Stress Injuries,Veterans Affairs CanadaDennis James, CAMHDonna Rogers, Four Counties Addictions Services Team(FourCAST)Donna Strawson, Community Mental Health, Community CareInformation Management (CCIM)Garth Martin, ConsultantGlenn Barnes, Dave Smith Youth Treatment CentreGloria Chaim, Child Youth and Family Program, Centre forAddiction and Mental HealthGlynis Burkhalter, Ray of Hope Youth Addiction ServicesHarry Whyte, Ray of Hope Youth Addiction ServicesHeather Bullock, Evidence Exchange Network (EENet), CAMHJai Mills, Central East Local Health Integration NetworkJan Hansen, Health Data Branch, Ministry of Health and Long‐Term CareJan Wighton, Data and Information Services, ConnexOntarioJanet Durbin, Provincial System Support Program, CAMHJanine Luce, Provincial System Support Program, CAMHJennifer Speers, ADAPTJoanna Henderson, Child Youth and Family Program, CAMHLinda Sibley, Addiction Services of Thames ValleyLois Alexanian, Maison FraternitéLucy Hume, Jean Tweed CentreMarianne Pope, Addiction Services of Thames ValleyMike O’Shea, North East Local Health Integration NetworkNancy Bradley, Jean Tweed CentreNandini Saxena, EENet, CAMHNila Sinnatamby, Ministry of Health and Long‐Term CarePatricia Syms Sutherland, Health System Transformation,Waterloo Wellington Local Health Integration NetworkPatty Chapman, South West Local Health IntegrationNetworkPaul McGary, Mental Health and Addictions Program,Lakeridge HealthPaul Welsh, Rideauwood Addiction and Family ServicesPeter Selby, Addictions Programs, CAMHRob Moore, Provincial System Support Program, CAMHRuth Stoddart, Ministry of Health and Long‐Term CareSarah Beland, Person with Lived ExperienceSusan Marshall, Community Mental Health CommonAssessment ProjectSusy Cannon, Jean Tweed CentreSuzanne Robinson, Central West Local Health IntegrationNetworkSylvie Guenther, Northern Ontario, CAMHWendy Prieur, North Bay Recovery HomeYing Jiang, Ministry of Health and Long‐Term Care5

FIRST STEPS TO ASSESSINGHEALTH EQUITYOn October 26, research staffparticipated in a day‐long retreat toconduct Health Equity ImpactAssessments (HEIA) for each of theDTFP assessment and performancemonitoring projects. HEIAs, a tooldeveloped by the Ministry of Health andLong‐Term Care, is intended to helpreduce inequalities that result frombarriers in access to quality healthservices and programming. It alsoinforms planning to increase positivehealth outcomes by identifying andmitigating unintended impacts of aninitiative prior to implementation. Theresults of the HEIA retreat are expectedto inform planning for next steps forthe DTFP projects. For moreinformation regarding HEIAs, visithttp://www.health.gov.on.ca/en/pro/programs/heia/.DTFP research team conducting Health EquityImpact AssessmentNEW MANDATORY DATA FIELDS AND DICTIONARYLed by Claudio Rocca, the DATIS Project has been reviewing literature and practices from across Canada– and around the world – to create a robust review of common data fields. This will enable the team tomake recommendations to the Health Data Branch of the Ministry of Health and Long‐term Care(MOHLTC) for revisions to the 66 mandatory fields that are currently being captured in Catalyst, abrowser‐based application that collects data from agencies. A review and revision of the fields will resultin a rich database of information that reflects current practices and needs.DATIS is also developing a Data Dictionary—an innovative tool that willcodify and explain each data element, creating consistency among allwho use and access the elements within Catalyst. In addition to clarifyingterms, the Dictionary will help identify gaps within the elements andbuild an accurate picture of the information that is being captured. Oncecompleted, the Dictionary will be available to all users of Catalyst.If you would like to make recommendations for revisions to the currentlycaptured mandatory fields, the DATIS team is accepting ideas untilMarch 2013. Contact your on‐site CSA (Catalyst Specialist Associate), whowill share your input with the team. Should the MOHLTC make changesto the current fields and tools based on the findings of the DTFPprojects, DATIS will play an integral role in supporting theseenhancements, while offering training in their use, and ongoing supportsto maximize their integration into Catalyst.“Maintainingconsistently high dataquality is an ongoingchallenge. We need amore consistentapproach, so to offeragencies astreamlined processfor data collection.”‐Claudio Rocca, DirectorDATIS6

EXPERIENCE MATTERS“Evidence” refers to more than just the scientific findings ofresearch studies; other kinds of knowledge can constituteevidence and inform the addictions treatment system. In additionto the Advisory Committee, Dr. Rush recently met with theOntario Systems DTFP Persons with Lived Experiences andFamilies Panel, hosted by Evidence Exchange Network (EENet).The panel brings together persons with lived experience ofmental health and addictions issues, and family members fromacross Ontario. It offers a rich source of knowledge and expertisethat often isn’t found in research studies.Dr. Rush’s meetings with the panel reflect a commitment toensuring that the experiences of diverse stakeholders have animpact on Ontario’s mental health and addictions system. Onekey learning that came out of the meetings, particularly relevant“The knowledge of peoplewith lived experience offersus a very valuableperspective. The mentalhealth and addictionssystem needs to hear whatthey have to say.”–Heather Bullock, Director,Knowledge Exchange, CAMHto the Client Perceptions of Care project, was the suggestion that clients receiving supports andservices may not always have a full sense of their options for care. As members of the panel pointedout, this may limit a person’s ability to offer his or her perceptions of care.FREQUENTLY ASKED QUESTIONSAm I allowed to use the CAMH Modified GAIN‐SS instrument?The original GAIN‐SS is copyrighted by Chestnut Health Systems. Ontario agencies canopt to use the CAMH‐modified version of the GAIN‐SS; however, licensing requirementswith Chestnut Health Systems still apply. For more information on the measure or licensure,please see www.chestnut.org/li/gain or email gainsupport@chestnut.org. In orderto monitor uptake, agencies that do opt to use the modified version are also asked toemail Gloria Chaim, co‐developer of the CAMH‐modified version, atGloria_chaim@camh.ca.Are agencies permitted to use the Ontario GAIN‐Q3 instruments?As with the CAMH‐modified GAIN‐SS instrument, the original GAIN‐Q3 is copyrighted byChestnut Health Systems. Agencies can opt to use the Ontario version; however, licensingand certification requirements with Chestnut Health Systems still apply. Discussionare underway with the Ministry of Health And Long‐Term Care regarding whether theOntario GAIN‐Q3 can replace some or all of the Admission and Discharge Criteria andAssessment Tools (ADAT) tools, mandatory for all publicly funded addictions agencies inthe province.For more information on the measure, licensure, or certification, please seewww.chestnut.org/li/gain or email gainsupport@chestnut.org. To obtain a copy of theOntario GAIN‐Q3, please email April Furlong at april.furlong@camh.ca.7

AgencyDTFP PILOT SITESExecutive Director/Study Lead(s)DTFP ProjectSite LeadCosting SARM* CS**Across Boundaries Aseefa Sarang Aseefa Sarang XADAPT Jennifer Spears Jennifer Spears XAddiction Centre Cate Sutherland Mary Boyce (SARM),X XAngie Degooyer (Costing),Cate Sutherland (SARM/Costing)Addiction Services of Thames Valley (ADSTV) Linda Sibley Megan Neill (SARM/CS),X X XMarianne Pope (Costing)Canadian Mental Health Association ‐ Kenora Anita Webb Anita Webb XCanadian Mental Health Association ‐HaltonRadhikaMarie HoyXSubramanyanCanadian Mental Health Association, Grey Bruce Claude Anderson Claude Anderson XDave Smith Youth Treatment Centre Glenn Barnes Sue O’Rielly XFour Counties Addiction Services Team Donna Rogers Alison Stagg (SARM & CS) X XG & B House Dan Purdon Linda Mowat (Costing),Dan Purdon (CS/Costing)XXGrey Bruce Health Services Jeff Franks Jeff Franks XHopeGrey Bruce Mental Health and Addiction Sandy Stockman Sandy Stockman XServicesJean Tweed Treatment Centre Nancy Bradley Julia Bloomenfeld,XSonali SagareMaison Fraternité – Fraternity House Denis Boileau Lois Alexanian (CS/Costing), XXDenis Boileau (Costing),Sylvie R. Carrie (Costing)Manitoulin Community Withdrawal Management Barb Deschamps Kathy Martin (CS),X XServicesRob Wabegijig (SARM)Nipissing Detoxification and Substance Abuse Dale McKinnon Kristine Lefebvre (CS),XXProgramsSteffi Wittmaack (Costing)North Bay Recovery Home Wendy Prieur Dave Pearson,XMarlene ThomasPine River InstituteVaughan Dowie, Laura MillsXJohn FallisPinewood Centre Paul McGary Rehana Suleman X XPortage Ontario Jennifer Blunt Bill Price XRay of Hope Youth Addictions Services Harry Whyte Glynis Burkhalter (CS/Costing), XXDebbie Woodhall (Costing)Red Lake: Community Counselling & Addiction Marcia Scarrow Marcia Scarrow XServicesRideauwood Addiction and Family Services Paul Welsh Kerry Moloney (SARM/CS) X XSunnybrook HospitalSaulo Castel, AmyCheungKim Castellano.Catarina LemosChild Youth and Family Program, CAMH Gloria Chaim Kate Ross X* SARM = Screening, Assessment and Recovery Monitoring, ** CS = Client SatisfactionThese projects are led by Dr. Brian Rush, Group Head of the Health Systems and Health Equity Research Unit. His project team is locatedat the Centre for Addiction and Mental Health in Toronto. For more information about these and other Ontario DTFP projects, pleasevisit www.ontariodtfp.ca.The creation of this document was supported by Evidence Exchange Network (EENet; formerly OMHAKEN). EENet actively promotes the use of researchevidence in decision‐making by providing an infrastructure to connect research to mental health and addictions research stakeholders acrossOntario, develops targeted KT products and tools and supports interactive exchanges.X8