Local Evaluation of Children's Services Learning from the Children's ...

More documents

Recommendations

Info

provided by projects weakened the accuracy of this data set. Reflecting the views ofa number of evaluators a Wave One evaluator described how the impacts ofChildren’s Fund interventions on educational targets, namely measures ofeducational attainment and attendance rates, are more easily demonstrated thanother outcomes since they are relatively precise, concrete concepts, and thatstatistical data sets exist that can be drawn on for secondary analysis:There are also issues around availability of data and we’ve been quite reliant onbeing able to match up individual records, which services provide us with, withsecondary data and we’ve only been able to do that for some areas ofChildren’s Fund sort of targets… they’re only able to look really at Children’sFund targets around educational attainment and inclusion [attendance].A related issue, as was suggested in one local evaluation report, there hastraditionally been reluctance among practitioners and decision makers to accept thecredibility of outputs which are not what is often referred to as hard evidence, that isquantitative evaluation data, in the context of evaluative, and indeed other types ofresearch. It is perhaps therefore inevitable that attempts by some evaluators toattribute the work of the Children’s Fund by drawing on qualitative evidence, such asby eliciting the voices of service users, has not been viewed by some Children’sFund stakeholders as legitimate forms of evidence that demonstrate impacts. Oneevaluator, for example, described the problems of demonstrating to members of apartnership board how a Children’s Fund service had had an impact on the lives ofthose using it:It might just be some kind of headline stats they [the partnership board] wantreally, something quite quick that they can kind of take in… I know that thatdrop in [Children’s Fund service] is a lifeline for some of those parents thatattend there. They wouldn’t know what to do with their children if they didn’t goto that drop in. And I can kind of write that up but it’s not hard evidence.Sometimes our senior officers want just the hard evidence, just the stats reallywhich sometimes doesn’t really tell the whole story.Example of an evaluation of impacts attributed to school-based Children’sFund projectsThe local evaluation report of a Wave One Children’s Fund partnership clearlydemonstrates the challenges in attempting to attribute the Children’s Fund activity tooutcome indicators, in this case to those related to education. The evaluation tookthe five themes identified by the local Children’s Fund programme and attempted tomeasure outcomes linked to specified projects related to these. One overarchingdifficulty identified by the evaluation related to measuring the effects of a time-limitedChapter 2 25
initiative such as the Children’s Fund, as the evaluation report states: ‘… a number ofissues have been raised about the suitability of many of these indicators whenmeasuring a relatively short-term, preventive programme such as the X’s Children’sFund’.The evaluation assessed the impacts of two Children’s Fund-supported projects thatinvolved one-to-one work with specific pupils that were aimed to improve schoolattendance and attainment: the first, a project that focused on pupil attendance,aimed to ‘(build) supportive relationships with all those involved with individual pupilswho are showing poor attendance’. The second, a volunteer reading project, wasdescribed as providing ‘extra educational support for many children who have fallenbelow the expected reading age for their school year’. Although these projects weretargeted within a number of schools, the former fourteen, the latter sixty, theevaluators experienced a number of difficulties in attempting to apply quantitativemeasures when assessing their impact, particularly when applied to the readingproject. As they explain: ‘measuring the impact of this kind of work…is problematic,as it tends to be informal, non-curriculum based and thus difficult to quantify’.The evaluation encountered a number of difficulties in using quantitative data sets(school attendance and unauthorised absence and performance at Key Stages 2 and3) to measure outcomes of Children’s Fund activity locally. Local and national schoolattendance figures were compared; improvements between 2001 and 2003 werehigher locally than the national average. They were cautious, however, in suggestingthat this could be attributed to the local Children’s Fund projects as they also foundthat attainment levels remained at less than the national average during this time.The evaluation therefore acknowledges the difficulties with attributing improvementsto the work of X Children’s Fund despite the high number of schools that weretargeted and therefore focuses on examples of specific cohorts of pupils in specificschools potentially benefiting from X Children’s Fund.Using this comparison the evaluation finds ‘clear improvements’ in attendance duringthe time the project has been running. However, again caution is advised asdiscrepancies between ‘authorised’ and ‘unauthorised’ absences makes for a morecomplicated picture – when considered separately either one of these had actuallyincreased in some schools during this period. This cautionary note compounds thepersistent issue of attribution which the evaluators feel is not solved by focusing onindividual schools: ‘since these statistics are at school level…it is difficult to26Chapter 2
Page 1 and 2: RESEARCHLocal Evaluation ofChildren
Page 4 and 5: When citing this report please show
Page 6 and 7: ContentsExecutive SummaryIIntroduct
Page 8 and 9: Local Evaluation of the Children’
Page 10 and 11: 10. Local evaluators highlighted th
Page 12 and 13: 20. Some evaluators involved childr
Page 14 and 15: successes of partnerships’ work (
Page 16: 38. Conditions reinforcing effectiv
Page 19 and 20: Children’s Fund initiative, namel
Page 21 and 22: engaging with policy makers, practi
Page 23 and 24: starting points and timetables of d
Page 25 and 26: 1. Process evaluation: critical ass
Page 27 and 28: time commitments and pressures amon
Page 29 and 30: tantamount to auditing when origina
Page 31 and 32: to a certain extent, because I was
Page 33 and 34: acknowledge that findings presented
Page 35 and 36: methods that were seen as being app
Page 38 and 39: Chapter 2: Methods Adopted by Child
Page 40 and 41: place. In one Wave One evaluation t
Page 44 and 45: confidently attribute any trends in
Page 46 and 47: A Wave Two evaluation report descri
Page 48 and 49: • Recruitment of the researcher:
Page 50 and 51: drawn on by evaluators include semi
Page 52 and 53: Chapter 3: How Local Evaluation Sup
Page 54 and 55: participation and work towards the
Page 56 and 57: highlighted in Table 1). These incl
Page 58 and 59: of had an impact. So we used a case
Page 60 and 61: Similarly an evaluator of a Wave Tw
Page 62 and 63: Evaluators sometimes struggled to a
Page 64: 6. A number of factors inhibit eval
Page 67 and 68: A number of evaluations described h
Page 69 and 70: 4.2 Key Messages from Local Evaluat
Page 71 and 72: for staff, typically dedicated part
Page 73 and 74: Where partnership working was descr
Page 75 and 76: strategic changes in children’s s
Page 77 and 78: 9. Evaluators reported that some co
Page 79 and 80: The importance of providing finding
Page 81 and 82: National Evaluation of the Children

Local Evaluation of Children's Services Learning from the Children's ...

Create successful ePaper yourself

Delete template?

Save as template?