Tom Meehan & Jane Wallace Implementation of Outcomes Initiative ...

5 YEARS ON - EMERGING AND CONTINUINGISSUES IN THE IMPLEMENTATION OFOUTCOME MEASURESImplementation of Outcomes Initiative -Tom Meehan & Jane Wallace(University of Queensland and The Park, Centrefor Mental Health)Implementation of Outcomes Program in Qld commenced in2003 Communication and Service Readiness Development of an ‘Outcomes Information System’ (OIS) Staff Training Support post-trainingEvaluation Funded by Queensland Health Redesign of Information System (CIMHA)Implementation of Outcomes Initiative - Implementation of Outcomes Program commenced in 2003 Communication and Service Readiness Development of an ‘Outcomes Information System’ (OIS) Staff Training Support post-training Redesign of Information System (CIMHA)

PercentParticipation Rates(Dec 2003 – Dec 2007)New EposideEnd Episode1009080706050403020100Dec-03 Dec-04 Dec-05 Dec-06 Dec-07New Eposide 43 56 58 64 66End Episode 40 51 55 61 64Evaluation - OverviewDesign:Prospective pre-test/post-test design using repeated measuresQuestionnaire Data: Immediately prior to training (n=2448) Immediately following training (n=2296) 8 months post-training (n=741) 5 Years post-training (n= 464)Focus Group Data: 8 months post-training: Focus group discussions (n=44) with324 staff and 4 discussion groups with 29 consumers 5 years post-training: Focus group discussions (n=71) with522 staff and 6 discussion groups with 53 consumersProfessional GroupsOverall group (n=2448) vs. Follow-up group (n=464)Length of Time in Mental Health -Overall group (2448) vs. Follow-up group (464)Overall group (n=2448)Sample at 5 yrs (n=464)Overall group (n=2448)Sample at 5 yrs (n=464)Length of Time in Mental Health ServiLength of Time in Mental Health ServProfessional GroupsProfessional GroupsTraining GroupFollow Up GroupTraining GroupFollow Up Group4040OtherMedicalOtherMedical3030OTOTSocial WorkerSocial Worker2020PsychologistNursePsychologistNursePercent1000-4 years 10-14 years 20-24 years 30-34 years 50-54 years5-9 years 15-19 years 25-29 years 35-39 yearsPercent1000-4 years 10-14 years 20-24 years 30-34 years5-9 years 15-19 years 25-29 years 35-39 yearsMean score54321Changes in knowledge of outcome measuresPost-training training (n=2256), Follow-up at 8 months (n=741),Follow-up at 5 years (n=464)(1= Poor, 2= Fair, 3= Good, 4 = V.Good, 5= Excellent)Pre-training Follow up 8 mths Follow up 5 yearsKnowledge ofoutcome measuresUnderstanding ofguidelines forcollecting dataUnderstanding ofOISChanges at 5 yrs all significant at p = .001Understanding ofhow to use data inpracticeMean scoreConfidence in using consumer outcomemeasures (Adult)(1= not at all confident, 2=reasonably confident, 3=Confident, 4 V.Confident,5=Extremely confident)54.5Both significant at p

Confidence in using outcome measures(1= not at all confident, 2=reasonably confident, 3=Confident, 4 V.Confident, 5=Extremelyconfident)Mean scoreHoNOSCA CGAS SDQ54All significant at p

Focus group data -Consumer issuesMHI seen as a means of letting ‘doctor’ know how one isfunctioningPerceived as being too clinicalMore feedback from staffMore attention to the way that MHI is offeredMore room for ‘free-hand’ commentsMore information for consumersMakes one more depressed…Focus Group Data – Ongoing issuesStaff factors:• Workforce: staff shortages, staff-patient ratios, agency/casualworkforce, staff acting in positions• Poor leadership and supervision - haphazard uptake ininitiatives (including outcomes)• General lack of support from senior medical staff – poorunderstanding of the outcomes initiative -• A large proportion of staff see little or no value in outcomemeasurement- ‘a paper exercise’• No reward for completing measures – no ‘punishment’ for notcompleting them• Staff ability to assess clients (mental state exam, etc)Focus Group Data – Ongoing issuesSome conclusionsMeasures:• Main focus is on HoNOS• Lack of confidence in HoNOS – difficult to rate,wide variation in scores, validity and reliabilityissues,• All measures too clinically focused –• Effort devoted to using measures appropriatelyis questionable – use of glossary, etc• Staff attitudes and MHIWide variation in the completion of measures – evenwithin the same serviceCompletion of MHI remains problematicTrend towards a decrease in confidence in completingmeasures.Some improvement in attitude towards outcomemeasures. Staff are more acceptable of the outcomesinitiative – less fearful of data.After 5 years……Conclusions• Ongoing training for staff – locally driven• Ongoing information for consumers• Long term view – possibly 10 -15 years!The way forward….A key driver in the establishment of an outcomes program issupport from senior medical staff“How has this client changed on outcome measures since welast reviewed him/her?”Establishing a paid position devoted to promoting outcomes(eg. Townsville model).Establishing outcomes resource people within services (eg.Cairns model).Rewarding staff for completing measures – PDP??? Linking outcomes to funding