Local Evaluation of Children's Services Learning from the Children's ...

More documents

Recommendations

Info

Chapter 2: Methods Adopted by Children’s Fund LocalEvaluationsThis chapter reflects on the ways local evaluations have embraced multiple(quantitative and qualitative) methods and types of evidence in assessing differentaspects of Children’s Fund partnerships’ work. The chapter highlights the range ofmethods employed by evaluators to measure impacts of Children’s Fund projectsand other interventions, and the methods used to generate learning from processesincluding partnership working and commissioning. The methods used to gather datafrom children, young people and their families and other ways children and youngpeople have engaged in the evaluation process are then described.2.1 Embracing Multiple-Method ApproachesLocal evaluators tended to draw on multiple quantitative and qualitative data sourcesand analysis methods that reflected the different elements of partnerships’programmes they aimed to examine. Quantitative methods, sometimescomplemented by qualitative methods, were frequently adopted for assessing theimpacts of Children’s Fund project interventions in terms of successfully reachingtarget groups and having a positive impact on aspects of their lives. Qualitativemethods, sometimes supplemented by quantitative methods, were primarilyemployed for generating learning about the processes of Children’s Fund activitysuch as the effectiveness of partnership working, commissioning processes andapproaches to mainstreaming. This research strategy reflects a trend in the UK,referred to earlier in this report, towards broadening what is considered legitimateevidence in evaluative research, and in particular a move towards embracing andvaluing qualitative as well as quantitative data collection reflecting theacknowledgement that different methods are appropriate in evaluating differentaspects of organisations’ work (for example, Spencer, et al., 2004).A Wave One evaluator’s multi-strategy approach, for example, typified thisorientation through using both quantitative and qualitative methods differentially toevaluate varying aspects of the partnership’s work. This evaluator carried outstructured (quantitative) questionnaire surveys with strategic partnership membersand with service providers, together with value for money assessments alongsidemore qualitatively oriented methods such as focus groups, in-depth interviews withstakeholders and observations. The structured questionnaires were deemedappropriately confidential and non-intrusive to gather honest opinions andperspectives to effectively …measure Children’s Fund performance against EveryChild Matters outcomes and to gather further information about the Children’s Fundaims of prevention, partnership working and participation. Follow-up, qualitative in-Chapter 2 21
depth interviews were then carried out, mainly with service providers, allowing theevaluators to … delve deeper into the processes and outcomes (impact) of servicesand generate a rich collection of data for analysis. The direct observation of serviceprovision followed this and was regarded as an opportunity for evaluators …to backup information gathered from questionnaires and interviews… and to contextualiseanalysis within the service setting.Similarly, another Wave One evaluation used both qualitative and quantitativemethods to analyse processes at both strategic and operational levels. Qualitativeanalysis of key partnership documents such as planning and discussion documentsand records of partnership board meetings and other activities at strategic levelhelped evaluators …understand and describe the development of the Children’sFund in relation to the core aims and principles of the initiative. A more quantitativelyorientated analysis of survey data from a census of all service providers enabledlocal evaluators to establish a: … description of a range of processes and structuresunderpinning the planning and implementation of services within the Children’s FundFramework. The sections below elaborate further on methods adopted in evaluatingimpacts and processes such as those discussed above, and reflect on some of thechallenges local evaluators have experienced in attributing impacts to specificChildren’s Fund activities.2.2 Measuring Impacts: the Attribution ProblemThe focus of measuring impacts varied between and within evaluation reports; manyexplored the impacts of project interventions on aspects of children’s livescorresponding to the Children’s Fund sub-objectives and the Every Child Mattersoutcomes. An extended exemplar of the work an evaluation of a Wave Onepartnership provided below illustrates the use of multiple quantitative and qualitativedata sources and methods at different levels for assessing impacts of Children’sFund projects on school attendance rates and educational attainment. It alsoillustrates some of the problems of demonstrating that outcomes are attributed to aparticular intervention.Some evaluators also explored impacts in terms of the extent to which the Children’sFund initiative was effectively targeting its services at specified user groups at a locallevel or as an assessment of the value for money of particular activities andinterventions. For these pieces of work service providers were often asked torespond to structured surveys to establish whether successful targeting had taken22Chapter 2
Page 1 and 2: RESEARCHLocal Evaluation ofChildren
Page 4 and 5: When citing this report please show
Page 6 and 7: ContentsExecutive SummaryIIntroduct
Page 8 and 9: Local Evaluation of the Children’
Page 10 and 11: 10. Local evaluators highlighted th
Page 12 and 13: 20. Some evaluators involved childr
Page 14 and 15: successes of partnerships’ work (
Page 16: 38. Conditions reinforcing effectiv
Page 19 and 20: Children’s Fund initiative, namel
Page 21 and 22: engaging with policy makers, practi
Page 23 and 24: starting points and timetables of d
Page 25 and 26: 1. Process evaluation: critical ass
Page 27 and 28: time commitments and pressures amon
Page 29 and 30: tantamount to auditing when origina
Page 31 and 32: to a certain extent, because I was
Page 33 and 34: acknowledge that findings presented
Page 35 and 36: methods that were seen as being app
Page 40 and 41: place. In one Wave One evaluation t
Page 42 and 43: provided by projects weakened the a
Page 44 and 45: confidently attribute any trends in
Page 46 and 47: A Wave Two evaluation report descri
Page 48 and 49: • Recruitment of the researcher:
Page 50 and 51: drawn on by evaluators include semi
Page 52 and 53: Chapter 3: How Local Evaluation Sup
Page 54 and 55: participation and work towards the
Page 56 and 57: highlighted in Table 1). These incl
Page 58 and 59: of had an impact. So we used a case
Page 60 and 61: Similarly an evaluator of a Wave Tw
Page 62 and 63: Evaluators sometimes struggled to a
Page 64: 6. A number of factors inhibit eval
Page 67 and 68: A number of evaluations described h
Page 69 and 70: 4.2 Key Messages from Local Evaluat
Page 71 and 72: for staff, typically dedicated part
Page 73 and 74: Where partnership working was descr
Page 75 and 76: strategic changes in children’s s
Page 77 and 78: 9. Evaluators reported that some co
Page 79 and 80: The importance of providing finding
Page 81 and 82: National Evaluation of the Children

Local Evaluation of Children's Services Learning from the Children's ...

Create successful ePaper yourself

Delete template?

Save as template?