Local Evaluation of Children's Services Learning from the Children's ...

More documents

Recommendations

Info

place. In one Wave One evaluation this approach was deemed an appropriatemethod as opinions could be gathered directly from the source of provision:‘Respondents were asked to think about the services their project provides and thegroups of individuals using those services in order to identify which level(s) ofprevention project activity is oriented towards’. An example of an attempt to measureimpact by focusing on value for money can be identified in a Wave Three evaluation.Assessing the value for money of Children’s Fund projects involved comparing thecosts of providing services with the numbers of interventions and numbers of serviceusers by drawing on monitoring returns data.Although the interpretation of ‘impact’ varied between local evaluators, many facedthe significant challenge of attempting to confidently attribute identified impacts to thework of specific Children’s Fund projects. As suggested earlier in this report theproblems of social exclusion are complex, multi-layered and interacting, making itdifficult to demonstrate causal links between a single intervention and a measurableoutcome (Percy-Smith, 2000; Pierson, 2002; Coote, et al., 2004). Writing aboutdevelopments in thinking about conducting social policy evaluation, Coulton notesthat the lives of children are complex and multi-layered: ‘…children and their familieslive within local communities…children interact with neighbours; participate in localinstitutions; receive social, health, and educational services’ (1995).Local evaluators of Children’s Fund partnerships described the difficulties theyexperienced in attempting to attribute impacts to the intervention of Children’s Fundprojects, especially in the context of the initiative’s preventive agenda. This suggeststhat the difficulties of attribution appear to be particularly acute in complex, multilayered,multi-agency community initiatives such as the Children’s Fund. Anevaluation of a Wave One partnership that reports on detailed work to explore theimpacts of selected project interventions concludes that, despite attempts to locaterelationships between the initiative and indicators such as school attendance andeducational attainment indicators, it had not been possible to ‘… confidently attributecurrent trends directly to the work of the X’s Children’s Fund’. Caveats relating towork on impacts such as these are also evident in other local evaluation reports. Forexample a large-scale quantitative evaluation of preventative interventionscommissioned by a Wave One partnership carried out extensive analysis of data setsincluding quarterly project monitoring data, postcode data, Indices of MultipleDeprivation and school performance data in order to demonstrate changes in schoolattendance and achievement. Although these statistics suggested that there hadChapter 2 23
een notable improvements in school outcomes, the report was careful to point outthat ‘… causation can not be attributed to the impact of the X Children’s Fund’.One related difficulty reflects the limited timescale of the Children’s Fund initiative inthe time-limited nature of local evaluators’ work. For example, a local evaluator of aWave One partnership described how the longer-term nature of prevention ofchildren and young people’s social exclusion contradicts the short-term, snapshotnature of local evaluation: … we wouldn’t have any specific indicators with thatparticular project because it’s a long term project and we won’t know the results formany years to come, and Children's Fund won’t be here. Similarly, an evaluator for aWave Three partnership pointed out the problems attributing impact to projects forwhich children may not be involved over a sustained period: … because it’s a rollingprogramme and children dip in and out… that is difficult to evaluate, you won’t get thesame child sort of continually going to the same project, it’s a moving sample.Other problems in assessing impacts reported by evaluators relate to the nature ofthe outcome measures and the data available to demonstrate impacts. Someevaluators felt that it was difficult to respond to the Children’s Fund sub-objectivesand Every Child Matters outcomes frameworks since they tend to define outcomes inrelatively broad terms that are potentially open to a range of interpretations, and thatestablished statistical data sets tend not to be available that clearly show change.This point was also noted in a report by the NECF (2004a) that explores the use ofindicators in assessing the impact of Children’s Fund activities. Illustrating this view aWave One partnership evaluator suggested:… at the moment the whole thing is set up obviously kind of aiming vaguely atthe kind of Every Child Matters outcomes or the original… the original CYPUoutcomes or objectives… But many of those are so global and so broad thatindividual projects can’t hope to demonstrate achievement against those andthey have got to be helped to break that right down into small steps. That is stilla major issue and I still find that there’s a great deal of confusion even atproject director level sometimes there’s confusion between outcomes andoutputs.Linked to this issue, a number of evaluators also described Children’s Fund quarterlymonitoring data as the principle source of information for assessing project impacts,in terms of the deployment of budgets and the numbers and characteristics of thechildren and young people using the services. However, difficulties related both tothe ability of projects to provide this information and to the usefulness of these datato local evaluators. In particular inconsistencies and unsubstantiated estimates24Chapter 2
Page 1 and 2: RESEARCHLocal Evaluation ofChildren
Page 4 and 5: When citing this report please show
Page 6 and 7: ContentsExecutive SummaryIIntroduct
Page 8 and 9: Local Evaluation of the Children’
Page 10 and 11: 10. Local evaluators highlighted th
Page 12 and 13: 20. Some evaluators involved childr
Page 14 and 15: successes of partnerships’ work (
Page 16: 38. Conditions reinforcing effectiv
Page 19 and 20: Children’s Fund initiative, namel
Page 21 and 22: engaging with policy makers, practi
Page 23 and 24: starting points and timetables of d
Page 25 and 26: 1. Process evaluation: critical ass
Page 27 and 28: time commitments and pressures amon
Page 29 and 30: tantamount to auditing when origina
Page 31 and 32: to a certain extent, because I was
Page 33 and 34: acknowledge that findings presented
Page 35 and 36: methods that were seen as being app
Page 38 and 39: Chapter 2: Methods Adopted by Child
Page 42 and 43: provided by projects weakened the a
Page 44 and 45: confidently attribute any trends in
Page 46 and 47: A Wave Two evaluation report descri
Page 48 and 49: • Recruitment of the researcher:
Page 50 and 51: drawn on by evaluators include semi
Page 52 and 53: Chapter 3: How Local Evaluation Sup
Page 54 and 55: participation and work towards the
Page 56 and 57: highlighted in Table 1). These incl
Page 58 and 59: of had an impact. So we used a case
Page 60 and 61: Similarly an evaluator of a Wave Tw
Page 62 and 63: Evaluators sometimes struggled to a
Page 64: 6. A number of factors inhibit eval
Page 67 and 68: A number of evaluations described h
Page 69 and 70: 4.2 Key Messages from Local Evaluat
Page 71 and 72: for staff, typically dedicated part
Page 73 and 74: Where partnership working was descr
Page 75 and 76: strategic changes in children’s s
Page 77 and 78: 9. Evaluators reported that some co
Page 79 and 80: The importance of providing finding
Page 81 and 82: National Evaluation of the Children

Local Evaluation of Children's Services Learning from the Children's ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?