Local Evaluation of Children's Services Learning from the Children's ...

More documents

Recommendations

Info

Similarly, a Wave One partnership report highlights that evaluation aims to identifywhat works and how, how these relate to targets, and that findings are expected toinform decision making beyond the Children’s Fund initiative:The local evaluator is the ‘critical friend’ of the Children’s Fund programme,providing the local agencies with information about what works, what actualimpact the projects have had, and how this impact came about. Monitoring theprogress towards national as well as local objectives and targets is an importantaspect of local evaluation. The evaluator is required to focus on the processes,outputs and short term outcomes of Children’s Fund services and should beable to influence the short term direction and thrust of local services, almost inan action research model.Two further roles were apparent in local evaluators’ work: firstly, some evaluationshave tended to adopt consultant/technician/analyst roles (see Table 1) by which theyhave carried out specific tasks defined by partnerships such as gathering andanalysing baseline data at some stages in the evaluation. These activities includemapping of activities introduced by a partnership or more broadly carrying outbaseline surveys of existing services relevant to children and young people in a LocalAuthority area, or the mapping of socioeconomic contexts that have informed earlydecisions on commissioning. These tasks indicate that some evaluators hadrelatively limited scope to define their own methods, interpret data and evaluatepartnerships’ activities. Secondly, a number of local evaluations have adoptedcapacity building roles (often also referred to as empowerment evaluation) in whichthey have sought to support partnership stakeholders, frequently project officers, toundertake evaluation activities themselves. The latter is detailed more fully inChapter 2 of this report.It is therefore evident that evaluations have often adopted multiple roles that changein emphasis over time, including varying degrees of capacity-building. Someevaluators also referred to changes stemming from policy developments at nationaland local levels that have influenced their abilities to undertake different roleseffectively; these are described below.1.2 Factors Influencing the Evaluation ProcessRelations with stakeholdersLocal evaluators highlighted a number of engagement and methodological problemsthey encountered. Local evaluators described a broad problem of engaging withservice users through projects, and of ‘research fatigue’ from stakeholder groups withresponses being fewer than anticipated. Some reports pointed to the problems ofChapter 1 9
time commitments and pressures among project providers and strategicstakeholders, limiting their engagement in the evaluation. It was noted by a WaveThree evaluator, for example, that the bureaucratic requirements imposed uponservice providers by the initiative made them reluctant to ask more of them:… the amount of paperwork being asked of people and the amount of tasksthey were being asked to do in order to get relatively small amounts of moneyfrom the Children’s Fund, were overburdening….we really felt that imposingmore on them, either in terms of asking them to collect stuff for us...or providinga lot of time for interviewing…really wasn’t fair.One methodological issue some local evaluators raised related to the perception oftheir role by different stakeholders. Whilst there was an understanding thatevaluators should generate learning to inform partnerships and projects’ decisionmaking, there was the more difficult perception, especially amongst serviceproviders, that local evaluation was analogous to an audit of their activities. Indeed, anumber of local evaluators described encountering service providers who wereuncertain about the purpose of evaluation and considerable time was spentconvincing them of its benefits. For example, a local evaluator suggested:I invested a lot of time in just going to see them, reassuring them, beingabsolutely open with my interview questions and giving them the schedule,giving them the information once I’d written it up. Not for them to change orcomment, but just to look at it, to say this is what I’m going to do. And that wasalso quite time consuming as well, as I remember.In some cases there were concerns that service providers were offering only positiveaccounts to local evaluators as they were connecting evaluation to futurecommissioning decisions. Indeed, this concern was sometimes justified as evaluatorswere asked by Children’s Fund programmes to provide information that coulddetermine the future of particular services. This may have resulted in the reluctanceof service providers to report more problematic aspects of their services, suspectingthat this could threaten future funding. As an evaluator suggested: Honest evaluationgets confused with the PR that is necessary to show that I’m [a service provider]doing this and we need on-going funding. Similarly, a report indicated that: ‘TheChildren’s Fund is both a funding and development body and providers areunderstandably wary of sharing too many difficulties and problems with the fundersfor fear of losing their funds’. An evaluator of a Wave Two partnership said: Whatwas also slightly inhibiting was that at that particular point the service providersweren’t used to being evaluated, and some almost saw it as the same as beinginspected, and so were very cagey. Reflecting this issue many local evaluation10Chapter 1
Page 1 and 2: RESEARCHLocal Evaluation ofChildren
Page 4 and 5: When citing this report please show
Page 6 and 7: ContentsExecutive SummaryIIntroduct
Page 8 and 9: Local Evaluation of the Children’
Page 10 and 11: 10. Local evaluators highlighted th
Page 12 and 13: 20. Some evaluators involved childr
Page 14 and 15: successes of partnerships’ work (
Page 16: 38. Conditions reinforcing effectiv
Page 19 and 20: Children’s Fund initiative, namel
Page 21 and 22: engaging with policy makers, practi
Page 23 and 24: starting points and timetables of d
Page 25: 1. Process evaluation: critical ass
Page 29 and 30: tantamount to auditing when origina
Page 31 and 32: to a certain extent, because I was
Page 33 and 34: acknowledge that findings presented
Page 35 and 36: methods that were seen as being app
Page 38 and 39: Chapter 2: Methods Adopted by Child
Page 40 and 41: place. In one Wave One evaluation t
Page 42 and 43: provided by projects weakened the a
Page 44 and 45: confidently attribute any trends in
Page 46 and 47: A Wave Two evaluation report descri
Page 48 and 49: • Recruitment of the researcher:
Page 50 and 51: drawn on by evaluators include semi
Page 52 and 53: Chapter 3: How Local Evaluation Sup
Page 54 and 55: participation and work towards the
Page 56 and 57: highlighted in Table 1). These incl
Page 58 and 59: of had an impact. So we used a case
Page 60 and 61: Similarly an evaluator of a Wave Tw
Page 62 and 63: Evaluators sometimes struggled to a
Page 64: 6. A number of factors inhibit eval
Page 67 and 68: A number of evaluations described h
Page 69 and 70: 4.2 Key Messages from Local Evaluat
Page 71 and 72: for staff, typically dedicated part
Page 73 and 74: Where partnership working was descr
Page 75 and 76: strategic changes in children’s s
Page 77 and 78:
9. Evaluators reported that some co
Page 79 and 80:
The importance of providing finding
Page 81 and 82:
National Evaluation of the Children
show all

Local Evaluation of Children's Services Learning from the Children's ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?