Local Evaluation of Children's Services Learning from the Children's ...

RESEARCHLocal Evaluation ofChildren’s ServicesLearning from the Children’s FundThe National Evaluation of the Children’s FundResearch Report RR783

Research ReportNo 783Local Evaluation ofChildren’s ServicesLearning from the Children’s FundThe National Evaluation of the Children’s FundThe views expressed in this report are the authors’ and do not necessarily reflect those of the Departmentfor Education and Skills.© The University of Birmingham 2006ISBN 1 84478 776 1

When citing this report please show authorship in full as follows:Spicer, N., and Smith, P. (2006)The National Evaluation of the Children’s Fund (NECF) ran from January 2003 toMarch 2006. A large number of people were involved in a variety of ways. Here welist members of the team who worked on either part-time or full-time bases during thethirty-nine months of the evaluation.NECF Team MembersThis report has been produced with the help of members of the Birmingham-based NECFresearch team.Lul Admasachew *David Allan*Professor Marian Barnes (Deputy Co-Director)Hanne BeirensAdrian BlackledgeLiz Chilton*Jane CoadLaura Day-AshleyProfessor Anne Edwards (Director)Ruth EvansTony FieldingRachel HekNathan HughesLucy LovelessNatasha MacNabAngus McCabe*Maggie McCutcheonKate Morris (Project Manager)Paul MasonKath PinnockGill PlumridgeHiromi YamashitaThe report draws on work undertaken by the whole team. Those who have particularlycontributed to producing it are marked by*.NECF can be contacted at –The National Evaluation of the Children’s FundGround FloorThe Watson BuildingThe University of BirminghamEdgbastonBirmingham B15 2TTTel0121 414 8089 – Amanda Owen-Meehan, Senior Project AdministratorEmail A.Owenmeehan@bham.ac.uk

AcknowledgementsNECF would like to thank the following people for the time they gave to theevaluation. It was much appreciated.• The children, young people and their families.• The practitioners who worked with the children and young people.• Local evaluators.• Members of partnership boards and other strategic groups.• The programme managers and their teams.• Regional managers and their teams.• NECF reference groups who gave on-going feedback as analyses developed.

ContentsExecutive SummaryIIntroduction 1Chapter 1Roles and Principles of Local Children’s FundEvaluations51.1 Roles of Children’s Fund Local Evaluations 51.2 Factors Influencing the Evaluation Process 91.3 Children’s Fund Local Evaluations’ Principles and Standards 121.4 Chapter Summary 18Chapter 2 Methods Adopted by Children’s Fund Local Evaluations 212.1 Embracing Multiple-method Approaches 212.2 Measuring Impacts: the Attribution Problem 222.3 Evaluating Processes 282.4 Engaging Children and Young People in Aspects of LocalEvaluations292.5 Chapter Summary 32Chapter 3How Local Evaluation Supports Children’s FundPartnerships’ Decision-Making353.1 Multiple Approaches to Disseminating Evaluation Findings 353.2 Partnerships’ Use of Evaluation Evidence in Decision-making 383.3 Factors Facilitating and Inhibiting the Use of EvaluationFindings413.4 Chapter Summary 45Chapter 4Prevention, Participation and Partnership Working: KeyMessages from Children’s Fund Local Evaluation Reports49

4.1 Key Messages from Local Evaluation: Prevention andPreventative Services4.2 Key Messages from Local Evaluation: Children and YoungPeople’s Participation49524.3 Key Messages from Local Evaluation: Partnership Working 544.4 Chapter Summary 58Conclusions 61References 63List of TablesTable 1 Roles and orientations of social policy evaluation 4Table 2Summary of the multiple roles adopted by Children’s Fundlocal evaluators over the course of partnerships’ development6

Local Evaluation of the Children’s Fund Initiative:Opportunities, Challenges and ProspectsThis report focuses on exploring four themes:• The roles, orientations and principles of local evaluators;• the evaluation methods that have been adopted by local evaluations;• the ways local evaluations have sought to support partnerships’ decisionmaking;• Key messages emerging from local evaluators’ reports and the ways localevaluators have sought to understand the underpinning themes of theChildren’s Fund initiative, namely, prevention, participation and partnershipworking.Executive summaryIntroduction1. There is a growing emphasis on evidence-based policy and practice in theUK. The Children’s Fund has fully embraced the importance of supportingpartnerships’ evidence based decision-making with local evaluation.2. Traditional experimental research and evaluation methods are now widelyseen as inappropriate for identifying what works in complex community-basedinitiatives such as the Children’s Fund where the relationships betweeninterventions and outcomes can be very complicated.3. The importance of drawing on qualitative as well as quantitative research andevaluation methods is widely recognised in some government departments. Itis also recognised that it may be appropriate to combine different evaluationmethods to address different questions in different settings. ‘Lay’ experiencesand perspectives such as service users’ views are now far more widely seenas legitimate forms of evidence than they were historically.4. In the past, evaluation has tended to be equated with auditing andperformance management. Evaluation is now often seen as generatingExecutive Summaryi

learning for policy makers and practitioners by providing formative or evendialogic forms of feedback.5. A number of factors and conditions influence the impact of research andevaluation on policy and practice. These include perceptions of the credibilityof evaluation, the ways evaluation findings are presented and disseminated,the relevance and usefulness of findings and the external pressures on policymakers and practitioners to act on findings.Roles and principles of local Children’s Fund evaluations6. The Children’s Fund Guidance establishes that partnerships shouldcommission local evaluations of their programmes of work, although variationsreflect individual partnership’s needs and priorities. Hence, local evaluators’roles have tended to correspond with partnerships’ starting points (in terms ofwave, for example), timetables of development, and to social andgeographical contexts. Some evaluators have been commissioned toundertake whole programme evaluations, whilst others focus on particularaspects of programmes’ work or selected activities and projects.7. The range of roles adopted by local evaluators includes generating learningabout processes intended to support partnerships’ strategic planning, ongoingmanagement and development, and providing an evidence base that informsthe development of mainstream preventative strategies and children andfamilies’ services.8. Evaluators have also aimed to measure impacts/outcomes of Children’s Fundactivities in terms of how sub-objectives have been addressed, how theinitiative has impacted on the lives of service users and how effectively targetgroups have been reached. This is described by evaluators as forming anevidence base on which partnerships are able to make decisions, rather thatsimply as a summative record of performance. This material is also intendedto inform decisions about projects and activities in terms of whether tocontinue to fund or promote for mainstreaming.9. Some evaluators have actively supported projects and programmes’capacities to undertake evaluation and monitoring activities themselvesthrough, for example, providing training and designing ‘toolkits’.iiExecutive Summary

10. Local evaluators highlighted their experience of a number of engagement andmethodological problems. Some evaluators indicated that time commitmentsand pressures among project staff and strategic stakeholders influenced theirabilities to participate in the evaluation.11. One methodological issue stems from the roles many local evaluators have asboth generating learning to develop more effective practices, and adopting arole regarded by some stakeholders as analogous to auditing Children’s Fundprojects. Indeed, some local evaluators encountered service providers whowere uncertain about the purpose of evaluation and considerable time wasspent convincing them of its benefits. Some concerns were raised that serviceproviders only offered positive accounts of their progress rather than revealingproblematic issues since they connected evaluation to decisions about futurecommissioning, a concern that was sometimes justified.12. Local evaluation is undertaken in a complex, changing policy environment.Uncertainties relating to the evolution of local level structures such aschildren’s trusts were described by a number of evaluators as problematic, aswas the announcement of budget cuts for the Children’s Fund initiative whichfor some undermined a systematic approach to conducting local evaluation.13. Local evaluators point to a number of principles and standards that haveguided them. These are: to provide useful and relevant material forpartnerships; to be responsive to change; to offer independent, balancedaccounts of partnerships’ activities; to be inclusive to different groups ofstakeholders; to use credible methods and evidence to support evaluationfindings; to adopt ethical approaches to evaluation.Methods adopted by Children’s Fund local evaluations14. Local evaluators drew on multiple quantitative and qualitative data sourcesand analysis methods that reflected the different elements of partnerships’programmes they aimed to examine. Quantitative methods, sometimescomplemented by qualitative methods, were frequently adopted for assessingthe impacts of interventions. Qualitative methods, sometimes supplementedby quantitative methods, were primarily employed for generating learningExecutive Summaryiii

about processes such as the effectiveness of partnership working,commissioning and approaches to mainstreaming.15. Quantitative methods adopted by evaluators include structured questionnairesand interviews with stakeholders and analysis of secondary statistical datasources, including the analysis of project monitoring data. Qualitative methodsdrawn on by evaluators include semi-structured or unstructured interviewswith stakeholders, focus groups, observations of project activities and analysisof partnership documents.16. Evaluators have aimed to measure the impacts of Children’s Fund projectsand other interventions on aspects of children’s lives corresponding to theChildren’s Fund sub-objectives and the Every Child Matters outcomes. Someevaluators also explored impacts in terms of how effectively projects targetedtheir services at user groups or as an assessment of the value for money ofparticular projects and other activities.17. There are a number of challenges to measuring impacts attributable tospecific Children’s Fund programmes and projects. The problems of socialexclusion tackled by the Children’s Fund are complex, multi-layered andinteracting, making it difficult to demonstrate causal links between a singleintervention and a measurable outcome. A related difficulty is the limitedextent to which time-limited local evaluation is able to identify longer-termoutcomes for children using Children’s Fund services.18. Some evaluators had difficulties responding to the Children’s Fund subobjectivesand Every Child Matters outcomes frameworks which defineoutcomes in relatively broad terms that are potentially open to a range ofinterpretations. Established statistical data sets tend not to be available thatclearly show change, although those relating to educational attainment andattendance are widely available. Some evaluators also noted the problems ofdrawing on inaccurate or inconsistent monitoring data provided by projects.19. Such challenges have led to some evaluators embracing the use of multiplemethods and multiple data sources creatively in ways that enable them tosuggest with some confidence that Children’s Fund projects have led topositive outcomes for children using the services.ivExecutive Summary

20. Some evaluators involved children and young people as questionnairerespondents or participants in interviews and/or focus groups; many used‘participatory’ methods of engaging children and young people. A minority oflocal evaluators involved children more fully through evaluation design,participation in fieldwork, analysis of data and presenting findings. This wasseen as making the questions evaluators asked more relevant to children andyoung people and helping to generate richer evaluation data.21. In some cases the lack of resources available to evaluators, difficultiesestablishing representative samples of those accessing services and the lowpriority given by the partnership to this type of activity limited the waysevaluators were able to engage children and young people.How local evaluation supports Children’s Fund partnerships’ decision making22. Evaluators have each adopted a range of summative, formative, and forsome, dialogic approaches to engaging with partnerships reflecting theirmultiple roles at different stages in partnerships’ development. Evaluationmaterial is disseminated through reports, presentations and workshops,websites, and some evaluators attend partnership board meetings, subgroupsor other organisational structures. A number of evaluators emphasised theimportance of ongoing dialogue with strategic partners about theirdevelopment rather than simply providing summative evaluation outputs.23. The importance of adopting appropriate means of feeding back evaluationmessages to a range of stakeholders including strategic partners, serviceproviders and children, young people and their families who used Children’sFund services was recognised by some evaluators. Some produced ‘childfriendly’ versions of reports or disseminated findings at events that involvedchildren and young people. However, it would appear that few evaluatorsprioritised service users, including children and young people, as an importantaudience of evaluation findings.24. Many local evaluations developed approaches to raising the capacity ofChildren’s Fund projects to self-evaluate their work through means such asworking alongside project staff in developing appropriate self-evaluation‘tools’, or through providing training.Executive Summaryv

25. A number of local evaluation reports identified where their recommendationshave been developed into tangible changes in the partnership. Others wereuncertain about how evaluation material had or would influence thepartnership. Evaluations appear to be offering material to partnerships to beused instrumentally, that is, through changing their practices and/or decidingwhich projects to re-commission. Local evaluation also appears to be usedconceptually in terms of expanding partnerships’ understandings relating toprevention, partnership working and participation (that is an enlightenmentmodel of the relationship between evaluation and decision making).26. Many programme managers participating in the NECF programme managers’questionnaire survey indicated that local evaluation helped identifysuccessful/less successful projects (51%) and how they worked well/less well(56%). 43% indicated local evaluation had helped decision-making aboutcontinuing funding projects and promoting projects for mainstreaming (39%).43% of programme managers indicated that local evaluation had helped thepartnership reflect on strategic practices and how to improve them. 47% ofprogramme managers indicated that local evaluation had helped to developthe partnership’s thinking about prevention, partnership working andprevention or as a means of helping to promote changes in practices andcultures among mainstream agencies (39%). Only 27% of programmemanagers suggested that local evaluation had been of limited value to thepartnership.27. Evaluators suggested that they could have more influence on partnerships byproviding findings that are relevant, useful, timely, accessible to a range ofstakeholders, and provide realistic recommendations. The importance ofengendering a relationship in which the partnership recognises the value ofevaluation and is open to both positive and more critical feedback was alsonoted.28. A number of factors inhibit evaluators’ influence on partnerships’ decisionmaking. Pressures from ongoing change nationally and within local authorityareas as mainstream agencies embrace a number of alternative, andsometimes, diverging agendas means that local Children’s Fund evaluationmay get drawn on selectively to legitimise decisions or to demonstrate theviExecutive Summary

successes of partnerships’ work (that is a politically/tactically driven model ofthe relationship between evaluation and decision making).29. Some evaluators indicated that they experienced difficulties controlling thedissemination of evaluation outputs, how outputs would be used, and indeedhad experienced difficulties reporting on negative aspects of partnerships’work. In some cases Children’s Fund board members’ multiple priorities andtheir commitments to their own agencies and other partnerships meant theyhad insufficient time to act on evaluation findings.30. A number of evaluators experienced disagreement with stakeholders aboutthe legitimacy of different evaluation methods. Some stakeholders favoured‘hard’ quantitative evidence and rejected the value of qualitative methods,whilst other appeared to have limited understandings of notions such asindicators and impacts or had unrealistic expectations of what evaluationscould produce. Maintaining ongoing dialogue in order to manage partnerships’expectations is therefore important.Prevention, participation and partnership working: key messages fromChildren’s Fund local evaluation reports31. The focus of local evaluators’ work on prevention has been to analyse theactivities of Children’s Fund projects. Most evaluations conceptualisedprevention in relation to the Children’s Fund sub-objectives, and/or to theEvery Child Matters outcomes framework in assessing projects’ progress. Anumber of evaluators related the work of projects to the four ‘tiers’ ofprevention adopted in the Children’s Fund Guidance.32. Local evaluators widely report on the beneficial impacts of Children’s Fundpreventive projects on children and young people’s lives and that manyprojects are effective at targeting ‘hard to reach’ groups of children and youngpeople. Some evaluators suggest that voluntary and community organisationshave been particularly successful in engaging and addressing the needs oftraditionally excluded groups through offering alternative approaches to thoseof the mainstream agencies in terms of being flexible, accessible and nonstigmatising.Executive Summaryvii

33. Understandings of ‘prevention’ among strategic stakeholders and serviceproviders within partnerships tend to be described in local evaluation reportsas divergent, although some acknowledge that progress is being made bypartnership boards in negotiating a consensus of understanding.34. Evaluators identified diverse definitions of participation among stakeholders inmost partnerships. Many evaluators reported that professionals and adults’definitions of participation tend to predominate, rather than those of childrenand young people themselves. Often these equate to consulting children andyoung people in order to inform the development of responsive services.Despite considerable commitment to participation, in practice progress tendsto be relatively limited, with few examples of children and young people’sdirect participation in decision making at strategic and project levels.35. A number of evaluators suggested that ‘hard to reach’ children and youngpeople and younger children were particularly under-represented in Children’sFund participative activities, although some evaluators identified examples ofparticipative activities that had been effective in engaging an inclusive rangeof children and young people.36. Local evaluators pointed to the importance of partnerships resourcingdedicated participation officers, the need to train practitioners and servicemanagers and the importance of providing organisational support for serviceproviders to develop effective participative practices. Considerable investmentof time is also needed if participation strategies are to be effective. Someevaluations suggest that voluntary and community organisations may beparticularly effective in promoting participation due to their abilities to work inflexible and innovative ways.37. Evaluators reported different experiences and understandings of partnershipworking, although within many partnerships agreements had been reached onwhat partnership working means. Some stakeholders saw partnership boards’responsibilities as ensuring probity and demonstrating that they met CentralGovernment requirements. Others emphasised collaborative working betweenorganisations to bring about cultural change in children’s services.viiiExecutive Summary

38. Conditions reinforcing effective partnership working include: sharing a visionof the purpose of the initiative; transparency of decision making; clarity aboutstakeholders’ responsibilities; good communications between groups ofstakeholders including members of the partnership board, Children’s Fundstaff, service providers and service users.39. Evaluators reported that some collaborative arrangements and networksbetween Children’s Fund projects had been facilitated by partnerships,although many suggested that such arrangements are relatively unevenacross partnerships’ activities. Other collaborative arrangements and networkswere reported as having been established by the organisations themselves.40. A number of evaluators commented that statutory agencies tended to leadstrategically, whilst voluntary and community organisations tended to provideChildren’s Fund services. However, within some partnerships voluntary andcommunity organisations had been strategically influential, indeed, theChildren’s Fund had raised the profile of the sector. Some stakeholderspredicted, however, that the voluntary and community sector would bemarginalised in the next phase of integrating children’s services.Executive Summaryix

IntroductionAims, scope and methodsThere is a growing academic and policy literature on developments in thinking abouthow evaluation contributes to evidence based policy and practice. Traditional‘hierarchies’ of research and evaluation methods that endorse experimental andquantitative methods are being set aside in favour of combined quantitative andqualitative methods for evaluating complex social policy initiatives. The importance ofembracing the experiences, perspectives and accounts of all stakeholders includingservice users and children and young people, and making evaluation findingsaccessible to multiple audiences, not just professionals and academics, have beenimportant developments in thinking on evaluation. There is also a growingunderstanding of the complex ways evaluation may (or may not) be used by policymakers and practitioners.This report considers the local evaluation of the Children’s Fund, a national initiativethat fully embraces the importance of evaluation in supporting partnerships to assessand develop their programmes of work. The nature and complexity of the initiativedoes, however, present a number of methodological and practical challenges toevaluators. One challenge is the highly divergent strategies and practices required atthe local level, to tackle what are complex, multi-layered and long-term issues ofsocial exclusion. This makes it particularly difficult to assess the effectiveness of theinitiative using traditional evaluation methods (Edwards & Fox, 2005). Suchchallenges have resulted in pragmatic creativity in the ways local evaluations havesought to demonstrate outcomes and support Children’s Fund partnerships’ planningin a context of rapid national and local policy change. The aim of this report is toreview the range of approaches adopted by local evaluators as well asacknowledging some of the tensions and compromises that have characterised theirwork. The report focuses on exploring four major themes:• The roles, orientations and principles of local evaluators;• The evaluation methods that have been adopted by local evaluations;• The ways local evaluations have sought to support partnerships’ decisionmaking;• Key messages emerging from local evaluators’ reports and the ways localevaluators have sought to understand the underpinning themes of theIntroduction 1

Children’s Fund initiative, namely, prevention, participation and partnershipworking.This report is based on the analysis of local external evaluation reports that havebeen made available to the National Evaluation of the Children’s Fund (NECF) todate 1 and the analysis of detailed, semi-structured interviews conducted withselected local evaluators (n=10). Interviewees were drawn from NECF case studypartnerships together with a sample of members of the NECF Local Evaluators’Reference Group. The programme managers’ questionnaire survey conducted by theNECF was also drawn on to highlight some of the ways programme managers sawlocal evaluation being used. 2 The emphasis of the analysis presented in this report isto illustrate the diversity of approaches and methods employed by local evaluators 3rather than exhaustively catalogue all local evaluation activity in all 149 partnerships.Hence, it is not possible for this report to quantify the numbers of evaluators adoptingdifferent approaches and methods. Neither is it the intention of this report to assessthe quality of individual evaluations or particular evaluation reports. To this endagencies/organisations conducting evaluations and individual Children’s Fundpartnerships are not identified to protect their anonymity.Developments in evaluation and evidence based policyIn the UK there is growing emphasis on the use of evidence in policy makingreflecting the current government’s pragmatic ideology of ‘what works’ by whichdecisions at national and local levels are expected to be founded on robust evidence.This has led to considerable expansion in evaluation infrastructure across the country(Solesbury, 2001). Nevertheless, it is recognised that key social problems such associal exclusion and health inequalities have multiple and interrelated facets andcauses including poverty, unemployment, family background, education,neighbourhood, opportunity and lifestyle. Hence, there can be no simple, causalassociation between a policy intervention and outcomes (Percy-Smith, 2000;Pierson, 2002; Coote, et al., 2004). Such recognition is reflected in complex, multiplelayeredpolicy responses to tackling social exclusion and the acknowledgement thata number of agencies would need to work in partnership to address these issues.1 At the time of writing reports relating to 80 partnerships were available for analysis.2 Based on 120 Children’s Fund programme managers’ responses (October-November,2005).3 The local evaluation reports represent the three Children’s Fund waves, regional spread,type of local authority (rural, urban, unitary, two-tier, metropolitan authorities) and agencyconducting the evaluation (university, consultant/consultancy, voluntary sector organisation).2Introduction

These shifts in thinking are widely embraced at local level by area-based initiativesincluding the Children’ Fund.This recognition has also led to changes in thinking about evaluation. In the UKclinical medicine has historically embraced the need for evidence based decisionmaking more than many other sectors, although it has tended to adopt a traditionalapproach to demonstrating ‘what works’. Standards of evidence in clinical medicineemphasise experimental methods; ‘randomised control trials’ and systematic reviewsof randomised control trials in particular are seen as the ‘gold standard’ of evidence(Becker & Bryman, 2004). Conversely, qualitative methods such as interviewingprofessionals or service users would be considered the least robust forms ofevidence. Whilst ‘lay’ perspectives such as the views of service users have nothistorically been valued, such experiences are now more readily acknowledged aslegitimate and important forms of evidence (Coote, et al., 2004; Becker & Bryman,2004). Traditional experimental approaches are now widely seen as inappropriate foridentifying what works in complex community and area-based initiatives in whichthere are no simple cause-and-effect relationships between interventions andoutcomes. A Cabinet Office report, establishing a framework for assessing qualitativeevidence, clearly signals that government departments are now valuing thecontribution of qualitative methods in social policy research and evaluation (Spencer,et al. 2004). Furthermore, it is widely recognised that a combination of evaluationmethods should be used appropriately to address different questions in differentsettings (Coote, et al., 2004; Becker & Bryman, 2004).There are also changes in the ways evaluators are seen as engaging with policymakers and practitioners. Rather than being equated with traditional auditing andperformance management roles, evaluation is frequently expected to provideformative or even dialogic feedback to help policy-makers and practitioners developprogrammes of activities. This is as well as generating more sophisticated,theoretically driven understandings of concepts such as social exclusion. Someevaluation is now committed to redressing social power imbalances and embracingdiverse and sometimes contradictory experiences, perspectives and accounts ofsocial problems and policy solutions (Donaldson & Scriven, 2003). Some of thesepotential roles and orientations identified in the literature are summarised in TableOne. Whilst some of these roles are well established in evaluation practice, othersare starting to emerge as important departures in the ways evaluation is seen asIntroduction 3

engaging with policy makers, practitioners and indeed other stakeholders of socialpolicies such as communities.It is also recognised in the literature that the extent to which research and evaluationinfluence policy depends on a range of factors. These may include: factors relating tothe credibility of evaluation such as data quality, perceptions among practitioners ofwhether evaluation methods are valid ways of generating knowledge and the extentto which evaluators are trusted; the ways in which evaluation findings are presentedand disseminated that are accessible, timely and meet the needs of the audience;the relevance and usefulness of evaluation findings to policy makers andpractitioners; and the abilities of policy makers and practitioners to act on evaluationfindings. These may be influenced by alternative and sometimes conflictingpressures that influence their decision making, such as changing policy directions atnational level, policy makers and practitioners’ abilities to interpret and apply findings,systems and institutional cultures of accepting and acting on evidence and policymakers and practitioners’ abilities to act on evaluation findings within budgetaryconstraints (McDonald, 2000; Davies, et al., 2000; Percy-Smith, et al., 2000; Becker& Bryman, 2004; Coote, et al., 2004).Table 1: Roles and orientations of social policy evaluationGeneratinglearning/informingorganisations’ developmentPerformance managementConsultant/technician/analystEmpowerment/capacitybuildingevaluationResearch-orientatedevaluationTheory-driven evaluationInclusive evaluationGenerating formative learning to helporganisations develop throughout their lives.Understanding what works and why; acting as acritical friendDemonstrating and showing accountability forimpact/success/failure of an organisation’sactivitiesProviding solutions to organisations’problems/questions; analysis of secondary data onbehalf of an organisationFacilitating organisations’ reflection and supportingthe capacity for self-evaluation/monitoringDeveloping and expanding practitioners/policymakers’ understandings of conceptsThe explicit use of a theoretical position thatinforms analysis/interpretation of evaluationmaterialAttempts to redress power imbalances andinequalities in society; sees communities asprinciple interest groups of evaluation4Introduction

Chapter 1: Roles and Principles of Local Children’s FundEvaluationsThis chapter describes the roles adopted by local evaluations of Children’s Fundpartnerships and reflects on a number of key factors that have influenced the abilitiesof local evaluators to undertake their roles at different stages in partnerships’development. The chapter then explores the range of principles adopted by localevaluators that were seen as reinforcing the credibility and effectiveness of theirwork.1.1 Roles of Children’s Fund Local EvaluationsThe Children’s Fund Guidance clarifies that local programmes are expected to beinnovative and develop new practices and approaches to tackle social exclusion forchildren and young people (CYPU, 2001). The lessons from the Children’s Fundprogrammes are to be elicited and the ways in which projects and other Children’sFund activities contribute to achieving outcomes for children are to be understood.The document indicates that local evaluators have the role of gauging theperformance of Children’s Fund activities and projects; this was expected to supportpartnerships to make appropriate changes to their programmes:The key purpose of local evaluation is to allow partnerships and services tobetter understand how well they are performing and to act on emergingevidence by making relevant changes to their programme. This will also ensurethat the development of the partnership and services is evidence-based.(CYPU, 2001, p25)Furthermore, it is aimed that local evaluation will identify key lessons from practice toinform local policy on children’s services:Local evaluation is to be commissioned by the partnership to enable you tounderstand what works, how and why; to monitor progress against local andnational objectives and targets; and to feed in lessons to reconfigure servicesas a result of evaluation findings… the overall purpose (of local evaluation)should be to inform future policy in relation to the local context and theparticular needs and circumstances of the community. The local evaluationshould consider whether services provided by the partnership are reachingyoung people, children and families showing early signs of difficulty and whateffect if any the services are having on their lives.(CYPU, 2001, p7)In practice, local evaluators have interpreted the aims defined in the Guidance andapplied aspects of them as outlined in their reports. The Children’s Fund initiative hasallowed for partnerships to develop strategies and practices according to theirrelevance at the local level. Reflecting this, local programmes have sought to definethe specific roles evaluators take on to correspond with programmes’ priorities,Chapter 1 5

starting points and timetables of development, and to social and geographicalcontexts and histories of developing services. Some local evaluators have thereforebeen commissioned to undertake whole programme evaluations looking at allaspects of the progress of a programme, whilst others have focused on particularaspects of programmes’ work or selected activities and projects. It is clear thatindividual evaluators have adopted a wide range of tasks; the range of tasks adoptedby a cross section of evaluations is illustrated in Table 2.Table 2: Summary of the multiple roles adopted by Children’s Fund localevaluators over the course of partnerships’ developmentPrinciple rolesKey tasksStrategic planning • Supporting a partnership by providing information andadvice that is drawn on in strategic planning• Providing baseline data such as identifying existingservices and activities, gaps in service provision; localsocioeconomic data; how the partnership fitsalongside existing statutory/VSC services in the localauthority area. Such activities are depicted asinforming early decisions such as commissioningprojects and other activities• Assessing the position of partnerships in relation tolocal and national policy changes in order to keeppartnerships up-to-date in a context of changing policydrawing on policy and research documentsOngoing management/development• Evaluating processes such as partnership working andchildren’s participation• Highlighting examples of successful/unsuccessfulpractices• Highlighting examples of good practice from outsidethe local authority area to inform programmes• Highlighting challenges and barriers to a partnership’swork• Evaluating specified themes or aspects ofprogrammes’ work• Looking at change over the course of a partnership’slife• Mapping partnership activities over time (for example,descriptions of the range of ways Children’s Fundservice providers embrace children and youngpeople’s participation)6Chapter 1

• Providing sets of recommendations to partnerships• Evaluating projects to inform ongoingcommissioning/funding of projectsInforming themainstream• Evaluating the Children’s Fund is seen as providing anevidence base that informs mainstream developmentof preventative strategies and children and familiesservicesCapacity building/evaluation support• Supporting projects (alongside programmes’ centralteams) in relation to evaluation and monitoringrequirements through one-to-one visits with projects,evaluation seminars and developing evaluation‘toolkits’. Evaluators have adopted training roles andhave facilitated projects exchanging practices aroundmonitoring and evaluation• Supporting projects to seek ongoing resources;assisting in the development of action plansMeasuringimpacts/outcomes• Assessing the extent to which (selected) Children’sFund activities are successfully achieving the subobjectivesand overall strategic objective of theinitiative at various stages in a partnership’sdevelopment• Assessing the impacts of projects/activities onservices users (children and young people, parentsand carers)• Assessing whether Children’s Fund projects/activitiesare effectively reaching specified groups of serviceusers• Assessing the extent to which partnerships have madean impact on the development on local children andfamilies’ services/development of strategic plans• Evaluating projects to inform the prioritisation ofongoing fundingLocal evaluators have typically sought to divide their work, or at least their approach,in terms of a consideration of processes and/or impacts. Illustrating this point, theevaluation of a Wave Three partnership, for example, clarifies how their work wasdivided according to these areas:Chapter 1 7

1. Process evaluation: critical assessment of the processes used to deliver theFund, mainly prevention, partnership and participation, but also strategic plansand the commissioning and monitoring processes.2. Impact evaluation: critical assessment of the impact the processes have hadon the children and families who access Children’s Fund services. Specificallythe impact that government envisions in the Children Act 2004Some evaluations placed more emphasis on generating learning about processes,while others placed more emphasis on evaluating impacts. Hence, local evaluationstend to correspond with both the generating learning/informing organisations’development and performance management roles outlined in Table 1.Many evaluators appear, however, to have adopted different roles over the course oftheir work. In the early stages of partnerships’ development, evaluators tended tofocus on providing feedback on processes such as commissioning and partnershipworking, and in doing so adopted formative forms of engagement with partnerships inwhich they supported early strategic planning and the ongoing management anddevelopment of programmes. As partnerships became more established manyevaluations switched their focus to producing summative outputs in which theymeasured impacts/outcomes. They also focussed on aspects of programmes whichhad been successful in meeting targets and sought to more systematically identifyareas of work which would be appropriate to continue to fund or to promote formainstreaming.The majority of evaluators described the role of evaluating impacts and that ofgenerating information to inform ongoing development of partnerships asoverlapping; the former is expected to form an evidence base for partnershipdecision making, rather than simply a summative record of performance. Illustratingthis overlap, an evaluation report of a Wave Three partnership emphasises theanalysis of partnership and project performance, findings from which form anevidence-base that is intended to inform the programme and strategic developmentof mainstream agencies:The aim of this local evaluation is to provide the partnership and projects withan analysis of current performance, and to use this understanding to develop astrong evidence base to make changes and improvements to the programme,which can also feed into mainstream provision and strategic development.8Chapter 1

Similarly, a Wave One partnership report highlights that evaluation aims to identifywhat works and how, how these relate to targets, and that findings are expected toinform decision making beyond the Children’s Fund initiative:The local evaluator is the ‘critical friend’ of the Children’s Fund programme,providing the local agencies with information about what works, what actualimpact the projects have had, and how this impact came about. Monitoring theprogress towards national as well as local objectives and targets is an importantaspect of local evaluation. The evaluator is required to focus on the processes,outputs and short term outcomes of Children’s Fund services and should beable to influence the short term direction and thrust of local services, almost inan action research model.Two further roles were apparent in local evaluators’ work: firstly, some evaluationshave tended to adopt consultant/technician/analyst roles (see Table 1) by which theyhave carried out specific tasks defined by partnerships such as gathering andanalysing baseline data at some stages in the evaluation. These activities includemapping of activities introduced by a partnership or more broadly carrying outbaseline surveys of existing services relevant to children and young people in a LocalAuthority area, or the mapping of socioeconomic contexts that have informed earlydecisions on commissioning. These tasks indicate that some evaluators hadrelatively limited scope to define their own methods, interpret data and evaluatepartnerships’ activities. Secondly, a number of local evaluations have adoptedcapacity building roles (often also referred to as empowerment evaluation) in whichthey have sought to support partnership stakeholders, frequently project officers, toundertake evaluation activities themselves. The latter is detailed more fully inChapter 2 of this report.It is therefore evident that evaluations have often adopted multiple roles that changein emphasis over time, including varying degrees of capacity-building. Someevaluators also referred to changes stemming from policy developments at nationaland local levels that have influenced their abilities to undertake different roleseffectively; these are described below.1.2 Factors Influencing the Evaluation ProcessRelations with stakeholdersLocal evaluators highlighted a number of engagement and methodological problemsthey encountered. Local evaluators described a broad problem of engaging withservice users through projects, and of ‘research fatigue’ from stakeholder groups withresponses being fewer than anticipated. Some reports pointed to the problems ofChapter 1 9

time commitments and pressures among project providers and strategicstakeholders, limiting their engagement in the evaluation. It was noted by a WaveThree evaluator, for example, that the bureaucratic requirements imposed uponservice providers by the initiative made them reluctant to ask more of them:… the amount of paperwork being asked of people and the amount of tasksthey were being asked to do in order to get relatively small amounts of moneyfrom the Children’s Fund, were overburdening….we really felt that imposingmore on them, either in terms of asking them to collect stuff for us...or providinga lot of time for interviewing…really wasn’t fair.One methodological issue some local evaluators raised related to the perception oftheir role by different stakeholders. Whilst there was an understanding thatevaluators should generate learning to inform partnerships and projects’ decisionmaking, there was the more difficult perception, especially amongst serviceproviders, that local evaluation was analogous to an audit of their activities. Indeed, anumber of local evaluators described encountering service providers who wereuncertain about the purpose of evaluation and considerable time was spentconvincing them of its benefits. For example, a local evaluator suggested:I invested a lot of time in just going to see them, reassuring them, beingabsolutely open with my interview questions and giving them the schedule,giving them the information once I’d written it up. Not for them to change orcomment, but just to look at it, to say this is what I’m going to do. And that wasalso quite time consuming as well, as I remember.In some cases there were concerns that service providers were offering only positiveaccounts to local evaluators as they were connecting evaluation to futurecommissioning decisions. Indeed, this concern was sometimes justified as evaluatorswere asked by Children’s Fund programmes to provide information that coulddetermine the future of particular services. This may have resulted in the reluctanceof service providers to report more problematic aspects of their services, suspectingthat this could threaten future funding. As an evaluator suggested: Honest evaluationgets confused with the PR that is necessary to show that I’m [a service provider]doing this and we need on-going funding. Similarly, a report indicated that: ‘TheChildren’s Fund is both a funding and development body and providers areunderstandably wary of sharing too many difficulties and problems with the fundersfor fear of losing their funds’. An evaluator of a Wave Two partnership said: Whatwas also slightly inhibiting was that at that particular point the service providersweren’t used to being evaluated, and some almost saw it as the same as beinginspected, and so were very cagey. Reflecting this issue many local evaluation10Chapter 1

eports tend to provide affirmation of progress rather than highlight less successfulaspects of project work. This approach may have meant the loss of important lessonslearnt from mistakes made, perhaps occurring more frequently where projects areseen as innovative or ground breaking.Evaluation in a context of changeEvaluators have undertaken their work in a complex, changing policy environment inwhich there have been different understandings locally of how the Children’s Fundshould be used. Policy changes in the lead up to the Children’s Bill/Act anduncertainties about financial management arrangements for the Children’s Fund (thechange in responsibilities from the Children and Young Persons Unit to theDepartment for Education and Skills), were described by a number of evaluators asproblematic. Uncertainties relating to the ongoing evolution of local level structuressuch as children’s trusts were also described as potentially problematic, as was theannouncement of budget cuts for the Children’s Fund initiative, undermining asystematic approach to planning and conducting evaluations. Indeed, NECFevidence suggests that this experience appears to have parallels with the difficultiesChildren’s Fund service providers have in planning their activities in the longer term.Illustrating this point an evaluation report of a Wave Three partnership states:‘[funding cuts] led to the evaluation having to be conducted on a year-by-year basiswith no guarantee of continued funding. The continued uncertainty over funding foryear 3 will adversely affect the strategic planning of the evaluation’. The changeswere also seen as potentially further undermining building relationships with keystakeholder groups including service providers. As one local evaluator reporthighlighted: ‘Funding uncertainties experienced by many projects also meant thatthere was some ambivalence to be engaged in external evaluation when thereappeared to be little or no benefit’.A number of local evaluators reported that their roles and positions have beensubject to unanticipated change or, have found that such roles resulted in tensions.For example, a local evaluator was originally engaged with Children’s Fund projectsby providing facilitation and support to enable projects to learn, develop and improvefrom evaluation activities. At a later stage the same evaluator was called on to assistin commissioning decisions about future funding for projects based on their analysisof projects’ performance. This role change was described as exaggerating potentialdisengagement and mistrust from some project staff who saw evaluation asChapter 1 11

tantamount to auditing when originally it had been promoted as a tool fordevelopment. Local evaluators described how this motivated some projects toprovide positive information to help support future funding rather than a morecomprehensive representation of their successes and failures. One evaluatorexplained:I have to say, that at the beginning of the evaluation, it was just going to be aprogramme for the programme itself. But once the evaluation was almostcomplete, it was quite clear to me that it was then going to be used to auditservices, which was inhibiting. Because it puts one in a very difficult positionabout what level of information you put in, because it can be used as a rodagainst services. So that was quite difficult and quite ethically challenging.Conversely, it was suggested by some evaluators that a number of projects haverecognised that evaluation can be a useful tool for demonstrating effectiveness topotential funders. For example, an evaluator of a Wave One partnership suggested:… if they don’t have any evaluation material it makes it very difficult for us tocontinue the service because we can’t see what impact they are having on theyoung people they’re working with. And I think service providers kind ofunderstand that. So they do see a reason why they should be evaluating andalso to kind of show off really to people and develop some best practice in thearea.The same evaluator indicated that more positive relationships had been built withprojects over time through building clear understandings of the purpose andimportance of evaluation:We definitely have a better working relationship than we used to I suppose. Ithink that’s because I’ve been here for a few years now and the Children's Fundhas been going for a long time. And they kind of understand what’s expected ofthem more, they understand why we need to evaluate and why they need toevaluate as well.1.3 Children’s Fund Local Evaluations’ Principles and StandardsThe Children’s Fund Guidance makes a number of statements about the standardslocal evaluators are expected to adopt; these include the usefulness of evaluationsfor developing evidence based understandings of what works and why, as well asbeing ‘formative, open, independent and inclusive’ (CYPU, 2001, p11). This alsoreflects current government thinking on standards that evaluations of public policyshould adopt. A recent Cabinet Office report, for example, states that evaluationshould: contribute to advancing knowledge and understanding; be defensible interms of methods that are appropriate to the evaluation questions being addressed;12Chapter 1

conducted rigorously in terms of systematic, transparent data collection, analysis andinterpretation; and make credible claims that are well founded, plausible and basedon data gathered (Spencer, et al., 2004).A number of local evaluators clearly explicate the principles guiding the conduct oftheir evaluations in their reports. Other evaluation reports allude to their guidingprinciples and standards in the descriptions of their approaches and methodologies.The commonly stated principles are described below although it should be noted thatthere are considerable variations between different local evaluations reflecting theirparticular sets of aims.Usefulness and relevanceMany local evaluators placed significant emphasis on providing material that wouldbe useful and relevant to Children’s Fund programmes. The need is recognised forregular and frequent reporting from local evaluators to ensure that findings andrecommendations have not been made redundant due to a long time lapse in theirproduction. The importance of being useful is also reflected by the formative,developmental approach to relationships between evaluators and partnerships. Thisis reflected in practices such as providing material to inform programme planning, aswell as producing findings at relevant intervals in a partnership’s development inorder to help stakeholders to reflect on and adapt practices.As suggested above, documentary and interview analysis points to the ways localevaluators have responded to the needs of Children’s Fund programmes. A range ofapproaches to defining the scope and direction were apparent. For some evaluationsthe partnership closely defined the local evaluators’ roles and aspects of its work thatthe evaluator would examine, although most appeared to have some latitude togenerate research designs and employ methods of data gathering and analysis theyfelt were appropriate to address the specific questions and needs of partnerships.Other evaluations appear to have had greater latitude to define the aspects ofpartnerships’ work that they examined. For example, one evaluator of a Wave Twopartnership explained:Well it was a bit of both. The programme manager decided that she wanted asecond evaluation, and she wanted it to be more in-depth. So she gave me theremit in terms of, I want it to cover services and I want to know exactly whatthey’re doing and what they’re not doing. How it was executed was absolutelydown to me… I think although it was a free rein; I think that was an advantageChapter 1 13

to a certain extent, because I was able to employ the techniques that I thoughtwere best at that particular time.Many evaluators were engaged in detailed meetings with programme managers andmembers of management teams in the early stages of evaluations and at regularintervals in order to develop sets of questions to be addressed, to agreemethodologies, timescales, themes and elements of the partnership to be evaluatedand a programme of feedback. Some evaluators appear to recognise the potentialsignificance and benefits of adopting a participative approach to evaluation (that is,responding to the views of service users in shaping their evaluation work), whilst alsoresponding to the articulated needs of those stakeholders managing the partnership(in particular programme managers and management teams). One Wave Onepartnership, for example, held an event that involved local young people who wereasked to raise questions they would like to be addressed by the local evaluation.Chapter 2 of this report provides examples of participative approaches of some localevaluations.Responsiveness to changeThe majority of evaluation reports indicate that the aims and scope of evaluationswere defined in agreement with partnership members; most often the programmemanager and/or members of the management team. Many evaluations also stressedthe importance of adopting a flexible approach enabling them to respond to changingneeds of the partnership over time. This is seen as important in a rapidly changingnational and local policy environment (as noted in Section 1.2 above). Someevaluations emphasised the importance of regular catch-up meetings with keyprogramme staff to ensure the ongoing relevance of the evaluation. Otherevaluations adopted an integrated approach within partnership structures (such asbeing formal members of partnership sub-groups that focus on evaluation andlearning) that enabled them to interactively engage with partnership stakeholders.One Wave One evaluator indicated that ideas about evaluation were shared througha number of networking events. Such gatherings brought together evaluation officersfrom initiatives such as Sure Start, Children’s Fund and Special Education projects toform a Monitoring and Evaluation Group that aims to combine approaches toevaluation consistently within the local authority area.14Chapter 1

Independent, balanced accounts of partnerships’ activitiesRelatively few reports explicitly stated that a key principle of the evaluation was itsindependence. One report, for example, indicated that it offered an independent,outsider’s view and hence offered perspectives that practitioners may not have. Anumber of reports indicated that it was their intention to generate learning from boththe successes as well as the failures of a partnership and its projects, and to highlightboth intended and unintended consequences of partnerships’ work. Some reportsstated that they had adopted a ‘critical friend’ stance in their relations withpartnerships; that negative comments were seen as assisting partnerships toimprove practice rather than as a judgement of poor performance. For example, anevaluation report of a Wave One partnership states: ‘Negative comments, where theyare given, are intended to be constructive – in the spirit of the “critical friend”’. Manylocal evaluation reports included some criticism of the progress of programmes,offering explanations as to why progress had been slow, proposing possible solutionsto bring about future improvements. Where the role of critical friend appeared to workmost successfully was where local evaluators were sensitive to the complex policyenvironments programmes were operating within, and where key members of theChildren’s Fund programme were receptive to the benefits of critical appraisal andprepared to use those findings to bring about improvements.A number of evaluators described the potential problems of balancing the intention ofbeing responsive to the needs of partnerships, and maintaining their independence.An evaluator of a Wave Two partnership indicated that whilst she felt the evaluationwas in principle relatively independent: I think [there was] minimal input from thepartnership board as well, because then it gets very political and agenda-driven. AndI didn’t feel that phase two [project evaluation] was that way, there had been somedifficulties in managing this independence in practice in terms of providing messagesthat the partnership board required of the evaluation:… as an independent evaluator, it’s all great and well, wonderful words andeverything, but actually some people don’t understand that. They still think thatyou’re paid by the Children's Fund therefore you will find what they want. Andsometimes the partnership board think, well we pay for you, so you will findwhat we want you to find, in the way that we want you to find it right.Some reports also stressed the importance of evaluation work being open tochallenge or correction by partnerships, indeed, the stage at which an evaluationengages with a partnership is important in terms of how definitive findings areexpected to be. Some reports, particularly early and interim reports, for example,Chapter 1 15

acknowledge that findings presented should be considered provisional, and thatmore definitive conclusions are expected in later reports. Some evaluations aretaking steps to confirm and verify evaluation findings through means such asestablishing reference groups of strategic partners and children and young peoplethat review and comment on material generated by the evaluation.Inclusive to different groups of stakeholdersMost evaluators stated that they were drawing on the voices of different stakeholdersto comment on different aspects of a partnership’s work. Such stakeholders includeprogramme managers and members of Children’s Fund staff, both statutory and VSCrepresentatives of partnership boards and other decision making structures, serviceproviders, and importantly children and young people and parents and carers whoare engaged in Children’s Fund activities. It is also recognised that it is important toensure that all participants in an evaluation receive effective feedback (see Chapter 3of this report). As suggested in a Wave Three partnership evaluation report, it isimportant to take an inclusive approach: So people feel involved in the evaluationprocess not subject to it. A small number of reports state that evaluations seek tobalance the interests of various stakeholder groups, rather than seek to support anyone group of stakeholders. As a local evaluation report of a Wave One partnershipstates:The approach taken in the design, implementation and reporting of thisevaluation is that of a public interest rather than ‘partisan’ evaluation. Itaddresses itself primarily to the work that has been done in relation to thestated aims and objectives of the…Children’s Fund. Its role is neither toadvocate for any one group or organisation, nor to arbitrate within or betweenthem.Other evaluations have adopted inclusive and participative approaches, aiming toinvolve children and young people, parents and carers in the evaluation designand/or its practice, as well as interview/questionnaire/focus group respondents. Forexample, the specific aim stated by a Wave One local evaluator was to empowerthose potentially benefiting from the Children’s Fund by eliciting their experiences ofChildren’s Fund services:We chose to focus on stakeholders like children and their carers due to the factthat they are the direct users of the Children’s Fund projects. Here we intend toempower the beneficiaries of the Children’s Fund by giving them a chance toexpress their views on ‘what works’ and ‘what doesn’t’ regarding the projectsthat they have been taking part in.16Chapter 1

It is not possible to accurately quantify levels and types engagement of children andyoung people, parents and carers in local evaluations, although analysis of reportssuggests that around half do not involve these groups. Of the rest, most involve themas interview/questionnaire/focus group respondents. A minority of evaluations appearto be involving them in the design stages, undertaking data collection or analysingevaluation data. Nevertheless, there are some examples of local evaluations thathave extensively involved children and young people. One such evaluation of aWave Three partnership that conducted a piece of work on uptake of services,involved children and young people in recruiting the researcher, designing theevaluation, gathering data and analysing findings. See Chapter 2 of this report for areview of the means adopted by some evaluators to engage children and youngpeople in various aspects of evaluations.The credibility of methods and evidenceA number of evaluators refer to the importance of adopting a defensible evaluationdesign, that is the evaluation is conducted rigorously and systematically in terms ofmethods of data collection, analysis and interpretation and that methods adopted areappropriate in answering particular questions evaluations set out to address. Somereports stress the importance of making credible claims and ensuring these are fullysupported by the data. Other reports state that the local evaluations were based onthe openness and transparency of the evaluation process and the methods used interms of data collection, analysis and interpretation of data. Hence, messagescontained in evaluation outputs should be fully supported by evidence. A broadly heldview is that evaluation methods should reflect the particular aspects of a partnershipthat evaluations are examining; this is reflected in use of multiple methods localevaluations use across different aspects of their work. Other reports emphasise theimportance of a broad-based view of partnerships both in terms of examining asmany aspects of partnerships’ work as possible and situating this work within localand national policy contexts and academic literature. The ways evaluation methodshave been used in combination in examining aspects of partnerships’ work andaddressing particular questions are explored in Chapter 2 of this report.Ethical approaches to evaluationA number of evaluation reports allude to ethical considerations that have guided theconduct of gathering data. Such considerations include the importance of beingculturally and linguistically sensitive in working with diverse groups and to berespectful to evaluation participants. Some evaluators indicated that they had appliedChapter 1 17

methods that were seen as being appropriate to different groups of stakeholders;methods for working with children and young people, for example, would be verydifferent to those for interviewing strategic stakeholders. Other evaluators point to theimportance of building trust among stakeholders, in particular service providers andservice users as the basis of encouraging a willingness to commit time and provideaccurate information. An evaluation of a Wave Three partnership highlights thedanger of building false expectations among service providers, who might believethat participating in the evaluation may lead to the continuation of funding.1.4 Chapter Summary1. The Children’s Fund Guidance establishes that partnerships shouldcommission local evaluations of their programmes of work, although variationsreflect individual partnership’s needs and priorities. Hence, local evaluators’roles have tended to correspond with partnerships’ starting points (in terms ofwave, for example), timetables of development, and to social andgeographical contexts. Some evaluators have been commissioned toundertake whole programme evaluations, whilst others focus on particularaspects of programmes’ work or selected activities and projects.2. The range of roles adopted by local evaluators includes generating learningabout processes intended to support partnerships’ strategic planning, ongoingmanagement and development, and providing an evidence base that informsthe development of mainstream preventative strategies and children andfamilies’ services.3. Evaluators have also aimed to measure impacts/outcomes of Children’s Fundactivities in terms of how sub-objectives have been addressed, how theinitiative has impacted on the lives of service users and how effectively targetgroups have been reached. This is described by evaluators as forming anevidence base on which partnerships are able to make decisions, rather thatsimply as a summative record of performance. This material is also intendedto inform decisions about projects and activities in terms of whether tocontinue to fund or promote for mainstreaming.18Chapter 1

4. Some evaluators have actively supported projects and programmes’capacities to undertake evaluation and monitoring activities themselvesthrough, for example, providing training and designing ‘toolkits’.5. Local evaluators highlighted their experience of a number of engagement andmethodological problems. Some evaluators indicated that time commitmentsand pressures among project staff and strategic stakeholders influenced theirabilities to participate in the evaluation.6. One methodological issue stems from the roles many local evaluators have asboth generating learning to develop more effective practices, and adopting arole regarded by some stakeholders as analogous to auditing Children’s Fundprojects. Indeed, some local evaluators encountered service providers whowere uncertain about the purpose of evaluation and considerable time wasspent convincing them of its benefits. Some concerns were raised that serviceproviders only offered positive accounts of their progress rather than revealingproblematic issues since they connected evaluation to decisions about futurecommissioning, a concern that was sometimes justified.7. Local evaluation is undertaken in a complex, changing policy environment.Uncertainties relating to the evolution of local level structures such aschildren’s trusts were described by a number of evaluators as problematic, aswas the announcement of budget cuts for the Children’s Fund initiative whichfor some undermined a systematic approach to conducting local evaluation.8. Local evaluators point to a number of principles and standards that haveguided them. These are: to provide useful and relevant material forpartnerships; to be responsive to change; to offer independent, balancedaccounts of partnerships’ activities; to be inclusive to different groups ofstakeholders; to use credible methods and evidence to support evaluationfindings; to adopt ethical approaches to evaluation.Chapter 1 19

Chapter 2: Methods Adopted by Children’s Fund LocalEvaluationsThis chapter reflects on the ways local evaluations have embraced multiple(quantitative and qualitative) methods and types of evidence in assessing differentaspects of Children’s Fund partnerships’ work. The chapter highlights the range ofmethods employed by evaluators to measure impacts of Children’s Fund projectsand other interventions, and the methods used to generate learning from processesincluding partnership working and commissioning. The methods used to gather datafrom children, young people and their families and other ways children and youngpeople have engaged in the evaluation process are then described.2.1 Embracing Multiple-Method ApproachesLocal evaluators tended to draw on multiple quantitative and qualitative data sourcesand analysis methods that reflected the different elements of partnerships’programmes they aimed to examine. Quantitative methods, sometimescomplemented by qualitative methods, were frequently adopted for assessing theimpacts of Children’s Fund project interventions in terms of successfully reachingtarget groups and having a positive impact on aspects of their lives. Qualitativemethods, sometimes supplemented by quantitative methods, were primarilyemployed for generating learning about the processes of Children’s Fund activitysuch as the effectiveness of partnership working, commissioning processes andapproaches to mainstreaming. This research strategy reflects a trend in the UK,referred to earlier in this report, towards broadening what is considered legitimateevidence in evaluative research, and in particular a move towards embracing andvaluing qualitative as well as quantitative data collection reflecting theacknowledgement that different methods are appropriate in evaluating differentaspects of organisations’ work (for example, Spencer, et al., 2004).A Wave One evaluator’s multi-strategy approach, for example, typified thisorientation through using both quantitative and qualitative methods differentially toevaluate varying aspects of the partnership’s work. This evaluator carried outstructured (quantitative) questionnaire surveys with strategic partnership membersand with service providers, together with value for money assessments alongsidemore qualitatively oriented methods such as focus groups, in-depth interviews withstakeholders and observations. The structured questionnaires were deemedappropriately confidential and non-intrusive to gather honest opinions andperspectives to effectively …measure Children’s Fund performance against EveryChild Matters outcomes and to gather further information about the Children’s Fundaims of prevention, partnership working and participation. Follow-up, qualitative in-Chapter 2 21

depth interviews were then carried out, mainly with service providers, allowing theevaluators to … delve deeper into the processes and outcomes (impact) of servicesand generate a rich collection of data for analysis. The direct observation of serviceprovision followed this and was regarded as an opportunity for evaluators …to backup information gathered from questionnaires and interviews… and to contextualiseanalysis within the service setting.Similarly, another Wave One evaluation used both qualitative and quantitativemethods to analyse processes at both strategic and operational levels. Qualitativeanalysis of key partnership documents such as planning and discussion documentsand records of partnership board meetings and other activities at strategic levelhelped evaluators …understand and describe the development of the Children’sFund in relation to the core aims and principles of the initiative. A more quantitativelyorientated analysis of survey data from a census of all service providers enabledlocal evaluators to establish a: … description of a range of processes and structuresunderpinning the planning and implementation of services within the Children’s FundFramework. The sections below elaborate further on methods adopted in evaluatingimpacts and processes such as those discussed above, and reflect on some of thechallenges local evaluators have experienced in attributing impacts to specificChildren’s Fund activities.2.2 Measuring Impacts: the Attribution ProblemThe focus of measuring impacts varied between and within evaluation reports; manyexplored the impacts of project interventions on aspects of children’s livescorresponding to the Children’s Fund sub-objectives and the Every Child Mattersoutcomes. An extended exemplar of the work an evaluation of a Wave Onepartnership provided below illustrates the use of multiple quantitative and qualitativedata sources and methods at different levels for assessing impacts of Children’sFund projects on school attendance rates and educational attainment. It alsoillustrates some of the problems of demonstrating that outcomes are attributed to aparticular intervention.Some evaluators also explored impacts in terms of the extent to which the Children’sFund initiative was effectively targeting its services at specified user groups at a locallevel or as an assessment of the value for money of particular activities andinterventions. For these pieces of work service providers were often asked torespond to structured surveys to establish whether successful targeting had taken22Chapter 2

place. In one Wave One evaluation this approach was deemed an appropriatemethod as opinions could be gathered directly from the source of provision:‘Respondents were asked to think about the services their project provides and thegroups of individuals using those services in order to identify which level(s) ofprevention project activity is oriented towards’. An example of an attempt to measureimpact by focusing on value for money can be identified in a Wave Three evaluation.Assessing the value for money of Children’s Fund projects involved comparing thecosts of providing services with the numbers of interventions and numbers of serviceusers by drawing on monitoring returns data.Although the interpretation of ‘impact’ varied between local evaluators, many facedthe significant challenge of attempting to confidently attribute identified impacts to thework of specific Children’s Fund projects. As suggested earlier in this report theproblems of social exclusion are complex, multi-layered and interacting, making itdifficult to demonstrate causal links between a single intervention and a measurableoutcome (Percy-Smith, 2000; Pierson, 2002; Coote, et al., 2004). Writing aboutdevelopments in thinking about conducting social policy evaluation, Coulton notesthat the lives of children are complex and multi-layered: ‘…children and their familieslive within local communities…children interact with neighbours; participate in localinstitutions; receive social, health, and educational services’ (1995).Local evaluators of Children’s Fund partnerships described the difficulties theyexperienced in attempting to attribute impacts to the intervention of Children’s Fundprojects, especially in the context of the initiative’s preventive agenda. This suggeststhat the difficulties of attribution appear to be particularly acute in complex, multilayered,multi-agency community initiatives such as the Children’s Fund. Anevaluation of a Wave One partnership that reports on detailed work to explore theimpacts of selected project interventions concludes that, despite attempts to locaterelationships between the initiative and indicators such as school attendance andeducational attainment indicators, it had not been possible to ‘… confidently attributecurrent trends directly to the work of the X’s Children’s Fund’. Caveats relating towork on impacts such as these are also evident in other local evaluation reports. Forexample a large-scale quantitative evaluation of preventative interventionscommissioned by a Wave One partnership carried out extensive analysis of data setsincluding quarterly project monitoring data, postcode data, Indices of MultipleDeprivation and school performance data in order to demonstrate changes in schoolattendance and achievement. Although these statistics suggested that there hadChapter 2 23

een notable improvements in school outcomes, the report was careful to point outthat ‘… causation can not be attributed to the impact of the X Children’s Fund’.One related difficulty reflects the limited timescale of the Children’s Fund initiative inthe time-limited nature of local evaluators’ work. For example, a local evaluator of aWave One partnership described how the longer-term nature of prevention ofchildren and young people’s social exclusion contradicts the short-term, snapshotnature of local evaluation: … we wouldn’t have any specific indicators with thatparticular project because it’s a long term project and we won’t know the results formany years to come, and Children's Fund won’t be here. Similarly, an evaluator for aWave Three partnership pointed out the problems attributing impact to projects forwhich children may not be involved over a sustained period: … because it’s a rollingprogramme and children dip in and out… that is difficult to evaluate, you won’t get thesame child sort of continually going to the same project, it’s a moving sample.Other problems in assessing impacts reported by evaluators relate to the nature ofthe outcome measures and the data available to demonstrate impacts. Someevaluators felt that it was difficult to respond to the Children’s Fund sub-objectivesand Every Child Matters outcomes frameworks since they tend to define outcomes inrelatively broad terms that are potentially open to a range of interpretations, and thatestablished statistical data sets tend not to be available that clearly show change.This point was also noted in a report by the NECF (2004a) that explores the use ofindicators in assessing the impact of Children’s Fund activities. Illustrating this view aWave One partnership evaluator suggested:… at the moment the whole thing is set up obviously kind of aiming vaguely atthe kind of Every Child Matters outcomes or the original… the original CYPUoutcomes or objectives… But many of those are so global and so broad thatindividual projects can’t hope to demonstrate achievement against those andthey have got to be helped to break that right down into small steps. That is stilla major issue and I still find that there’s a great deal of confusion even atproject director level sometimes there’s confusion between outcomes andoutputs.Linked to this issue, a number of evaluators also described Children’s Fund quarterlymonitoring data as the principle source of information for assessing project impacts,in terms of the deployment of budgets and the numbers and characteristics of thechildren and young people using the services. However, difficulties related both tothe ability of projects to provide this information and to the usefulness of these datato local evaluators. In particular inconsistencies and unsubstantiated estimates24Chapter 2

provided by projects weakened the accuracy of this data set. Reflecting the views ofa number of evaluators a Wave One evaluator described how the impacts ofChildren’s Fund interventions on educational targets, namely measures ofeducational attainment and attendance rates, are more easily demonstrated thanother outcomes since they are relatively precise, concrete concepts, and thatstatistical data sets exist that can be drawn on for secondary analysis:There are also issues around availability of data and we’ve been quite reliant onbeing able to match up individual records, which services provide us with, withsecondary data and we’ve only been able to do that for some areas ofChildren’s Fund sort of targets… they’re only able to look really at Children’sFund targets around educational attainment and inclusion [attendance].A related issue, as was suggested in one local evaluation report, there hastraditionally been reluctance among practitioners and decision makers to accept thecredibility of outputs which are not what is often referred to as hard evidence, that isquantitative evaluation data, in the context of evaluative, and indeed other types ofresearch. It is perhaps therefore inevitable that attempts by some evaluators toattribute the work of the Children’s Fund by drawing on qualitative evidence, such asby eliciting the voices of service users, has not been viewed by some Children’sFund stakeholders as legitimate forms of evidence that demonstrate impacts. Oneevaluator, for example, described the problems of demonstrating to members of apartnership board how a Children’s Fund service had had an impact on the lives ofthose using it:It might just be some kind of headline stats they [the partnership board] wantreally, something quite quick that they can kind of take in… I know that thatdrop in [Children’s Fund service] is a lifeline for some of those parents thatattend there. They wouldn’t know what to do with their children if they didn’t goto that drop in. And I can kind of write that up but it’s not hard evidence.Sometimes our senior officers want just the hard evidence, just the stats reallywhich sometimes doesn’t really tell the whole story.Example of an evaluation of impacts attributed to school-based Children’sFund projectsThe local evaluation report of a Wave One Children’s Fund partnership clearlydemonstrates the challenges in attempting to attribute the Children’s Fund activity tooutcome indicators, in this case to those related to education. The evaluation tookthe five themes identified by the local Children’s Fund programme and attempted tomeasure outcomes linked to specified projects related to these. One overarchingdifficulty identified by the evaluation related to measuring the effects of a time-limitedChapter 2 25

initiative such as the Children’s Fund, as the evaluation report states: ‘… a number ofissues have been raised about the suitability of many of these indicators whenmeasuring a relatively short-term, preventive programme such as the X’s Children’sFund’.The evaluation assessed the impacts of two Children’s Fund-supported projects thatinvolved one-to-one work with specific pupils that were aimed to improve schoolattendance and attainment: the first, a project that focused on pupil attendance,aimed to ‘(build) supportive relationships with all those involved with individual pupilswho are showing poor attendance’. The second, a volunteer reading project, wasdescribed as providing ‘extra educational support for many children who have fallenbelow the expected reading age for their school year’. Although these projects weretargeted within a number of schools, the former fourteen, the latter sixty, theevaluators experienced a number of difficulties in attempting to apply quantitativemeasures when assessing their impact, particularly when applied to the readingproject. As they explain: ‘measuring the impact of this kind of work…is problematic,as it tends to be informal, non-curriculum based and thus difficult to quantify’.The evaluation encountered a number of difficulties in using quantitative data sets(school attendance and unauthorised absence and performance at Key Stages 2 and3) to measure outcomes of Children’s Fund activity locally. Local and national schoolattendance figures were compared; improvements between 2001 and 2003 werehigher locally than the national average. They were cautious, however, in suggestingthat this could be attributed to the local Children’s Fund projects as they also foundthat attainment levels remained at less than the national average during this time.The evaluation therefore acknowledges the difficulties with attributing improvementsto the work of X Children’s Fund despite the high number of schools that weretargeted and therefore focuses on examples of specific cohorts of pupils in specificschools potentially benefiting from X Children’s Fund.Using this comparison the evaluation finds ‘clear improvements’ in attendance duringthe time the project has been running. However, again caution is advised asdiscrepancies between ‘authorised’ and ‘unauthorised’ absences makes for a morecomplicated picture – when considered separately either one of these had actuallyincreased in some schools during this period. This cautionary note compounds thepersistent issue of attribution which the evaluators feel is not solved by focusing onindividual schools: ‘since these statistics are at school level…it is difficult to26Chapter 2

confidently attribute any trends in this data to the work of the X (project) as otherfactors may be affecting the figures’.The evaluation then analysed child level data based on twelve randomly selectedchildren in the schools targeted by the project focusing on school attendance. At thislevel the evaluation report appears able to identify and evidence significant impactsof the project work with more confidence. For example, one pupil’s attendance levelsincreased by 40% over a seven month period, during which s/he had receivedsupport from the project: ‘…it is clear that without the support and encouragement ofthe X (Project), all of these children and probably many others, would have missed alarger percentage of their early years education’. There were also attempts tomeasure the impacts of the reading project at individual child level by considering anational pupil survey conducted by the organisation operating the project. The surveymeasured impacts in terms of ‘attitudes to reading’ and demonstrated animprovement in this during 2002. Evaluators detailed the improved reading ages offive randomly chosen pupils at one of the targeted schools, as described by the staffdelivering the project. Qualitative impacts were also described. These were based onsome one-to-one interviews with children accessing the service and duringevaluation events where children were encouraged to express their thoughts andopinions on the projects they had been accessing through art work and writing.Analysis of such data led evaluators to note the development of ‘supportiverelationships built between the children and the project volunteers’ and ‘the effort… toincorporate the children’s views on how their work should progress and particularlywhat books they enjoyed’.It therefore appears that it was when the evaluator reported upon data gathered fromchild-level analysis that impacts could be described with greater confidence. Thesequalitative data typically referred to the experiences of individual children rather thanbeing of a quantitative nature. There remained, however, potential difficulties withthese data. For example, the selection of the five pupils by those providing theservice may not have resulted in a typical sample.Overall, however, the evaluators argued that their multiple methods approach(analysis of quantitative data at national, local, school and individual child-levels,together with qualitative interviews allowing for service providers’ and service users’perceptions of project impacts to be revealed) whilst imperfect, allowed them toChapter 2 27

suggest with some confidence that Children’s Fund projects had led to positiveoutcomes for children using the services.2.3 Evaluating ProcessesMany local evaluation reports undertook assessments of key Children’s Fundprocesses associated with the delivery of programmes including: setting targets andcommissioning services; processes supporting the infrastructure of the programme,including communications arrangements, financial management arrangements andcross project information and referral arrangements; processes in place to report onprogress, including monitoring and evaluation arrangements; work on establishingnew models of working among mainstream agencies; children and young people’sparticipation and the effectiveness of strategic partnership arrangements.A range of methods tended to be used by individual evaluators. A significant numberof reports describe the use of data elicited from in-depth qualitative interviews orfocus groups with strategic stakeholders when discussing perceptions andexperiences relating to the effectiveness of strategic processes developed by localChildren’s Funds. The focus for interviews was typically the experiences and learningindividual members of partnership boards could offer, with some evaluators carryingout interviews with stakeholders in the wider strategic landscape in the locality, forexample key individuals in Social Services or large voluntary organisations who maybe affected by Children’s Fund activity.Other methods adopted include observational work within strategic and in projectsettings, quantitative questionnaire surveys to establish the systems and strategies inplace between stakeholders and documentary analysis of key partnership planningdocuments and board minutes. A Wave Three evaluator, for example, described howdrawing on observational, interviews and focus group work at both strategic andoperational levels of Children’s Fund activity enabled a more detailed, more holisticpicture of partnership working, prevention and participation to emerge:I think it gave me a holistic picture of everything for a start because of courselike I say, the ‘three Ps’ [prevention, participation and partnership], I feel arealmost concepts as well as… they’re almost virtual… and people have differentunderstandings in different areas, and may have different understandings of it.So it’s about looking at their understanding, or their interpretation of preventionand participation, and partnership working. They gave me a more holisticunderstanding of that, and shaped that.28Chapter 2

A Wave Two evaluation report describes how initial questionnaire survey work withstrategic partners was followed by in-depth qualitative focus group research withthese stakeholders; this had allowed them to examine a broad range of themes indepth and explore further issues that emerged over the course of the evaluation andthereby gain a relatively complete understanding of key processes and theireffectiveness:… we looked at the composition of the Board; at the organisations represented.We looked at how Executive Board members were recruited on to the Boardand then looked at the relationships between the organisations on the Boardand how those had relationships with the Children’s Fund team. We looked athow the involvement of each organisation affected the ways of working andallocation of resources within the programme; we looked at the frequency andnature of meetings and at the input of children, young people and parents andcarers into that process; the effectiveness of communication and the extent towhich they had increased the capacity of the local community and voluntarygroups.2.4 Engaging Children and Young People in Aspects of Local EvaluationsMany evaluations have involved children and young people in their work in a range ofways. This has been particularly pertinent in the context of the Children’s Fundprogramme, which has emphasised the participation of potential beneficiaries ofChildren’s Fund services, namely children, young people and their families, but alsothose of communities more broadly. Indeed, such groups have been traditionallyexcluded from evaluation and as subjects of social policy research (for example,Coote, et al., 2004). The analysis of local evaluation reports suggests that less thanhalf of evaluations involved children and young people as questionnaire respondentsor participants in interviews and/or focus groups; of these many indicated that theyhad used ‘participatory’ methods of engaging children and young people. One localevaluator of a Wave Three partnership, for example, emphasised the importance ofcapturing the ‘felt’ impact of the programme on the lives of individual children, youngpeople and their families. This evaluation team attempted to do this by employing avariety of data collection techniques, designed to gather information in appropriateenvironments:… where appropriate, interviews with service users made use of a portablediary room, to gauge user’s opinions of the service they attend…the aim of thediary room was to provide a private, fun space in which users could give theirthoughts and opinions on the service they attend…the diary room was childfriendlyand safe.A Wave One partnership evaluation report indicates that as a result of work in whichchildren and young people’s voices were elicited, they were also able to collate ‘aChapter 2 29

ich source of data regarding perspectives about such issues as problems of dailylife, physical environments and the experiences of accessing support services’.Another evaluator suggested that by tailoring the methods to children and youngpeople, they would be more engaged and therefore more willing to express theirviews:I mean we’ve definitely chosen specific methodologies for children and youngpeople like the photos and drawing and the art and craft work because wewanted to make it fun really. We didn’t really want to give them another kind ofboring tick box questionnaire, so we’ve definitely tailored our methodologiesaround themA Wave One evaluation pursued a participatory approach in their work with childrenand young people. This involved various creative activities and pursuits that wereconsidered appropriate on a number of levels in terms of encouraging individualexpression, establishing appropriate targeting within services and empoweringservice beneficiaries:Wherever possible a participatory approach was developed, for example usingimages, artwork and games, to engage young people and staff incommunicating what they thought and felt about the activity they had beeninvolved in. This sort of activity went some way in identifying outcomes relatingto target children, along with children’s engagement. It was an approach thatmirrored the original rhetoric of the Children’s Fund Guidance in relation toempowering children, young people, families and communities to becomeinvolved in planning and delivering services, eventually taking responsibility andcontrol of solutions to problems for themselves.A minority of local evaluators appear to have involved children more fully throughevaluation design, participation in fieldwork, analysis of data and presenting findings.For example, a Wave Three evaluation drew on the Alderson and Montgomery modelof ‘participation in decision making’ whereby children and young people are includedin the evaluation process from the outset. The process had two stages; firstlyincluding children and young people in decisions relating to the design of theevaluation and then inviting them to have a significant input in how the research wasconducted. Similarly, a Wave Three evaluation used a participatory action researchmodel attempting to ascertain why children, young people and their families might notbe inclined to use Children’s Fund services in this partnership area. This involveddrawing on these potential service-users at all stages of the evaluation process, asdescribed below:30Chapter 2

• Recruitment of the researcher: children and young people were involved inwriting the job description, advertising post and interviewing candidates.• Design of the evaluation: steering groups of children, young people andparents helped design and test the evaluation approaches and fieldworkinstruments.• Fieldwork: children and young people undertook fieldwork with support andguidance.• Analyses of the findings: children helped to analyse and interpret datacollected and consider how to present the findings.• Presentation of the findings: children and young people were involved indisseminating findings.A similar approach was adopted by a Wave One evaluation. This evaluation workedwith the partnership to establish the ‘… questions they (children and young people)would want answered from an evaluation’. Following this, the intention was to workwith a small team of older young people to ‘select several questions and visitidentified projects to talk to staff and young people involved…about the work they aredoing’. What were called ‘Young Reporters’ collected evaluation material fromprojects in an attempt to draw children and young people into the evaluation process.According to the evaluators, a key advantage was that children and young peoplebrought informality to the research process and an ability to elicit appropriate data,the generation of which may not have otherwise been recorded, as is described inthe evaluation report:The Young Reporters are able, in a very direct way, to feedback to project staffwhat the children… are saying and also to ask challenging questions…Feedback from projects suggests that this is quite a disarming process thatinduces replies that might not have been so readily given to an adult evaluatorFor this and a number of other evaluations, recognising the skills of children andyoung people and applying these to research processes was seen as enrichingevaluation data. By being interviewed by their peers, younger service users wereable to ‘give candid views about their experiences’. Furthermore the success of theYoung Reporters could be used as an exemplar of good practice within theChildren’s Fund partnership and be drawn upon as a useful evaluation tool:It has been acknowledged within the programme that the Young Reportersinitiative represents a model of good practice…projects are learning from theChapter 2 31

example. Further efforts, such as a proposed good practice poster, should bemade to ensure that this model is effectively disseminated.Despite the positive experiences of involving children and young people in theevaluation processes, there were some limitations to this work. These relate to lackof resources available to evaluators, difficulties establishing representative samplesof those accessing services and the low priority given by the partnership to this typeof evaluation activity. Lack of adequate resources for substantial pieces of work withchildren and young people had potential consequences for important relationships oftrust. A Wave One evaluation, for example, described how limitations in evaluationfunding had undermined some aspects of their work:Due to the evaluation parameters and resources, it was only possible to makeone contact with the (young) advisers. While the input received wasdisseminated widely in the planning process, it was not possible to conductfurther consultation and feedback to the advisers.Furthermore, a Wave One evaluation faced difficulties when attempting to involvechildren with special needs in their work; this required evaluators to balance themethodological requirement for representation with the perceived complexity ofworking with children and young people characterised as having special needs: Wetried to make it as random as possible but I think inevitably given the nature of thework and in some cases the quite extreme difficulties that those children had, therewere definite sensitivities involved which we had to be constantly mindful of.2.5 Chapter Summary1. Local evaluators drew on multiple quantitative and qualitative data sourcesand analysis methods that reflected the different elements of partnerships’programmes they aimed to examine. Quantitative methods, sometimescomplemented by qualitative methods, were frequently adopted for assessingthe impacts of interventions. Qualitative methods, sometimes supplementedby quantitative methods, were primarily employed for generating learningabout processes such as the effectiveness of partnership working,commissioning and approaches to mainstreaming.2. Quantitative methods adopted by evaluators include structured questionnairesand interviews with stakeholders and analysis of secondary statistical datasources, including the analysis of project monitoring data. Qualitative methods32Chapter 2

drawn on by evaluators include semi-structured or unstructured interviewswith stakeholders, focus groups, observations of project activities and analysisof partnership documents.3. Evaluators have aimed to measure the impacts of Children’s Fund projectsand other interventions on aspects of children’s lives corresponding to theChildren’s Fund sub-objectives and the Every Child Matters outcomes. Someevaluators also explored impacts in terms of how effectively projects targetedtheir services at user groups or as an assessment of the value for money ofparticular projects and other activities.4. There are a number of challenges to measuring impacts attributable tospecific Children’s Fund programmes and projects. The problems of socialexclusion tackled by the Children’s Fund are complex, multi-layered andinteracting, making it difficult to demonstrate causal links between a singleintervention and a measurable outcome. A related difficulty is the limitedextent to which time-limited local evaluation is able to identify longer-termoutcomes for children using Children’s Fund services.5. Some evaluators had difficulties responding to the Children’s Fund subobjectivesand Every Child Matters outcomes frameworks which defineoutcomes in relatively broad terms that are potentially open to a range ofinterpretations. Established statistical data sets tend not to be available thatclearly show change, although those relating to educational attainment andattendance are widely available. Some evaluators also noted the problems ofdrawing on inaccurate or inconsistent monitoring data provided by projects.6. Such challenges have led to some evaluators embracing the use of multiplemethods and multiple data sources creatively in ways that enable them tosuggest with some confidence that Children’s Fund projects have led topositive outcomes for children using the services.7. Some evaluators involved children and young people as questionnairerespondents or participants in interviews and/or focus groups; many used‘participatory’ methods of engaging children and young people. A minority oflocal evaluators involved children more fully through evaluation design,participation in fieldwork, analysis of data and presenting findings. This wasChapter 2 33

seen as making the questions evaluators asked more relevant to children andyoung people and helping to generate richer evaluation data.8. In some cases the lack of resources available to evaluators, difficultiesestablishing representative samples of those accessing services and the lowpriority given by the partnership to this type of activity limited the waysevaluators were able to engage children and young people.34Chapter 2

Chapter 3: How Local Evaluation Supports Children’s FundPartnerships’ Decision-MakingThis chapter reviews the ways local evaluations have engaged with Children’s Fundpartnerships in seeking to support their decision making. The chapter highlights therange of ways evaluators have disseminated findings to partnerships and to whichgroups of stakeholders. Some of the key ways evaluation findings are used bypartnerships are then explored and the factors facilitating and inhibiting the abilities ofevaluators to influence partnerships’ decision making are identified.3.1 Multiple Approaches to Disseminating Evaluation FindingsIndividual evaluators have adopted a range of summative, formative, and for some,dialogic approaches to engaging with partnerships, reflecting their multiple roles atdifferent stages in partnerships’ development. Hence, evaluators have disseminatedmaterial to partnerships through writing reports (including interim reports, full reports,reports that focus on particular themes and those that examine project impacts andgood practices). They have also provided material through presentations andworkshops to partnership stakeholders, distributing material on websites, andthrough periodic attendance at Children’s Fund partnership board meetings,subgroups and other organisational structures.A number of evaluations emphasised the importance of creating a dialogue withstrategic partners in terms of developing how they might take things forward ratherthan providing summative evaluation outputs, as an evaluator of a Wave Twopartnership suggested:I think there’s definitely a dialogue. I mean I think we… and it’s a supportiveone. We don’t just kind of give information and then leave it; it’s about workingwith the partnership trying to say, we will feed it back but then do a little miniworkshop to get them to do some thinking around it, and about what that meansfor them and their partners and things. So it’s much more of a supportive kind ofrole really.Many evaluations sought to combine different approaches to engaging withpartnerships. For example a Wave One evaluator described how their adoption of anaction research model of working led to the dissemination of formative andsummative findings through ongoing discussions with the partnership. It was hopedthat such conversations would encourage the partnership to reflect on practices andhence be developmental:We do say ‘this is what you’ve done’ but, obviously, part of what we’re doing ismaking recommendations around how they might develop in the future andChapter 3 35

then trying to encourage discussion around that. In that sense, it is kind of anaction research type model, where we are making recommendations and tryingto facilitate discussion around those recommendations, as well as saying ‘this iswhat you’ve done’, sort of reflecting back practice to them.Another Wave One evaluation stressed the importance of breaking down boundariesbetween evaluation and practice; effective evaluation should engage in apartnership’s organisational and practice development:… our approach is action research and what that basically means is thatinstead of in a sense doing a systematic evaluation analysis of each individualproject or anything like that… we pretty much collapse the boundaries betweenwhat we call evaluation, organisational development, organisational learning,you know, practice development and policy development. In a sense they allbecome part of a learning system.Multiple audiences for evaluation findingsMany evaluators also pointed to the importance of adopting appropriate means offeeding back messages from the evaluation to a range of Children’s Fundstakeholders including strategic partners, service providers and children, youngpeople and their families who used Children’s Fund services. Local evaluationreports tended not to comment on their intended audience, although theirpresentation suggests they are intended for members of Children’s Fund programmepartnership boards. However, a relatively small number of local evaluators have alsoproduced ‘child friendly’ versions of reports and some local evaluators have providedversions of their reports for managers and practitioners of specific Children’s Fundprojects.One evaluator suggested that events had been an effective way of highlighting someof the issues raised by evaluation to a wider constituency of stakeholders. Thepartnership had run a children’s event to highlight key evaluation issues relating toprojects at which project workers, children, young people and their families attended,together with members of the partnership board, representatives from the children’strust and other strategic managers from statutory agencies attended. Events drawingthose working at strategic and operational levels together were also highlighted asbeing potentially effective for disseminating messages in a Wave One partnership.Although initially sceptical about the potential for limited interest among invitedprofessionals, the evaluator commented on the value of encouraging first-handexperiences of children and young people’s skills and abilities in an informal setting.It was suggested that such events can begin to develop strategic thinking around36Chapter 3

participation and work towards the reengagement between the strategic andoperational aspects of Children’s Fund activity:… we planned a huge fun day in effect, an hour of which was taken up with usdoing all sorts of work; asking questions and so on. Now my interpretation ofthat is that professionals would see the hour as being the bit that they should beat which they may or may not get to and the rest of it as being not work. Butactually those people that were there for the day came away thinking that they’dgot something quite special out of the process because they’d seen how thechildren had inter-acted across the whole day. They’d learnt something aboutchildren’s participation across the whole day and actually for some I think it wasan interesting process of re-engaging with children. I mean clearly some hadtheir own children and so on and others have gone in their own personal livestheir own children are grown up and so on. But my sense is that there is aseparation sometimes between children’s policy-makers and children.It would appear however that few evaluators prioritised service users, includingchildren and young people, as an important audience of evaluation findings. Forexample, one evaluator described how it was initially difficult to find the mostappropriate ways of engaging parents in evaluation feedback:[I asked] I just want to know what you think about it. Is that something that youcan relate to or…? Any of the recommendations are they completely out of thewater that you just think, what on earth are you talking about; don’t know whatthat’s about? And I think to be honest with you, people were quite afraid of theevaluation to begin with, like they couldn’t challenge it, or they couldn’t questionit. Or they couldn’t say, well what do you mean by that, what on earth does thatmean?Building partnerships’ capacities to undertake evaluationIn addition to effectively communicating findings to partnership stakeholders, manylocal evaluations described a key focus of their work as developing approaches toraising the capacity of Children’s Fund projects to self-evaluate their work. Someevaluators regarded this as central to the evaluation process, whether this wasthrough evaluators working alongside project staff in developing appropriate selfevaluation‘tools’ during the life of the evaluation, or through more formal trainingwithin workshops involving members of the partnership. For example, one evaluationsupported Children’s Fund projects to identify, summarise and record their successesand failures and future strategies. Another evaluator was specifically involved inbuilding the capacity of children and young people to evaluate Children’s Fundservices.For some evaluations developing self-evaluation tools contributed to a number ofother dissemination and supportive activities. A Wave One evaluator, for example,Chapter 3 37

developed self-evaluation tools through a series of workshops during the lifetime ofthe evaluation which aimed to offer ongoing support to projects developing theirabilities to evaluate themselves. Self-evaluation agreements were drawn up forprojects who were required to report on aims, indicators and their methods ofpreference towards drawing on data already available and complementing this withdata elicited from surveys, interviews and observations. The development ofevaluation tools was also a feature of an evaluation of a Wave Two partnership; akey aim was described as being to: ‘examine the processes by which theManagement Team collect data and monitor progress… how can evaluation beembedded in the process?’.A Wave Two evaluation was able to support those at the operational level throughfocus group work following the analysis of a survey on self-evaluation toolsdistributed to all service providers. Other evaluators were commissioned specificallyto carry out this role, as is described by a Wave One evaluator: …they’vecommissioned us, purely on a consultancy basis to write an evaluation tool kit fortheir project; a self-evaluation tool kit.The emphasis on developing self-evaluation practices has varied according to theneeds of the partnerships and available resources. A Wave One evaluator describedhow an attempt to carry out this work had been squeezed out as time and financialconstraints had taken over proceedings. This has been a cause of some frustrationamongst evaluators who may have regarded this aspect of the research as crucial,for example those describing themselves as action researchers. Conversely a WaveOne partnership described how concerns over future resources had positively driventhe development of self-evaluation tools, as those at strategic level sought toencourage service providers to consider their place beyond the life of the Children’sFund:It’s partly about an increasing focus on the mainstreaming agenda and the needto get projects to the point where they are able to pursue other forms of funding,etc. beyond the Children's Fund and the feeling that sustainability was partlyabout being able to demonstrate that they were achieving key outcomes.3.2 Partnerships’ Use of Evaluation Evidence in Decision MakingYoung, et al. (2002) propose a framework to describe the range of waysorganisations draw on evaluation findings to support their decision making processesin practice (as distinct from the intended roles or orientations of evaluations38Chapter 3

highlighted in Table 1). These include: knowledge-driven model (an organisation’swork is driven by evaluation); problem-solving model (evaluation is used to solveproblems/address specific questions/provide a knowledge base for decision making);enlightenment model (evaluation provides a greater understanding of concepts andissues) interactive-model (practice and evaluation interactively influence oneanother); political/tactical model (evaluation is drawn on selectively by anorganisation to legitimise decisions). It is also useful here to draw on the distinctionmade by Clarke (2001): evaluation findings tend to be used by organisations in twoways instrumentally and conceptually. The former denotes the ways organisationsmay act directly on evaluation findings in terms of, for example, changing practicesand deciding to fund particular activities; the latter denotes the ways evaluationmaterial may influence organisations’ understanding of concepts more broadly, andthereby informing strategies and strategic thinking.In practice there were mixed perceptions among Children’s Fund local evaluatorsrelating to the extent to which they believed that they had had an influence onpartnerships’ decision making, although for many the relationships betweenpartnerships and evaluators appear to correspond with the problem solving model(Young, et al. 2002) and evaluation findings used instrumentally (Clarke, 2001).Evidence derived from local evaluation reports and interviews with local evaluatorssuggest that the majority of evaluations appear to pragmatically gather and analysedata concerned with the impact of Children’s Fund programmes, or processes suchas children’s participation and partnership working with a view to offering materialthat is intended to be used by partnerships instrumentally, such as through changingpractices and deciding to re-commission particular projects. Indeed, manyprogramme managers participating in the NECF programme managers’questionnaire survey 4 suggested that local evaluation had helped to identifysuccessful/less successful projects (51%) and how they worked well/less well (56%),as well as to make decisions about continuing funding projects (43%) or whichprojects to promote for mainstreaming (39%). Many programme managers alsoindicated that local evaluation had helped the partnership reflect on strategicpractices and how to improve them (43%).There was also some indication that evaluation had offered partnerships newconcepts or broader frames of understanding (corresponding with the enlightenment4 Based on 120 Children’s Fund programme managers’ responses.Chapter 3 39

model and the notion of using evaluation conceptually). 47% of programmemanagers indicated that local evaluation had helped to develop the partnership’sthinking about prevention, partnership working and prevention or as a means ofhelping to promote changes in practices and cultures among mainstream agencies(39%).Importantly, few programme managers suggested that local evaluation had been oflimited value to the partnership (27%), indeed, a number of evaluation reportsidentified where their recommendations have been developed into tangible changesin the partnership, although others appeared not to be certain about how evaluationmaterial had or would influence the partnership. An example from a Wave Twoevaluation demonstrates how data gathered on Children’s Fund projects had had adirect impact on how the partnership directed project work, for example byrecommending the systematic collection of baseline data and the need to encourageinvolvement at project level, of children with special needs.The evaluation report suggests that perhaps the most significant impact of theevaluation data was its influence on decisions around future commissioning ofChildren’s Fund projects, as the following extract describes: ‘Findings from theevaluation informed the deliberations of the X Children’s Fund Appraisal Panel thatmet in December to consider the X Children’s Fund Continuation Funding Proposalsand Appraisal Records, 2004-2006’. Another evaluator, for example, pointed to theimportance of disseminating the issues children raised as part of the evaluation; thishad been particularly influential in terms of informing the partnership:There was work with children who had ADHD a project where they wereparticularly good on participation and they really did, they really kind of broughtwork through that really did involve children and you could see the kind of pridethat those children had in that work and that did, I believe anyway, have animpact on the partnership and also on the work of health professionals in thatarea.Another evaluator suggested that evaluation findings had informed the partnership ofsuccesses among projects; the partnership had drawn on these finding in makingdecisions about which project activities to promote for mainstreaming:… there were pieces of work that I think were mainstreamed in the endbecause of what we said; because the reports that we produced did evidencesuccess in a way that the partnership could understand and in a way which kind40Chapter 3

of had an impact. So we used a case study approach on work that we thoughtwas particularly good or of particular attention.3.3 Factors Facilitating and Inhibiting the Use of Evaluation FindingsA number of local evaluators pointed to factors that had facilitated them effectivelyinfluencing partnerships’ work. For example, an evaluation report of a Wave OneChildren’s Fund partnership defines a number of measures that are seen as enablingthe partnership to respond to evaluation findings:• information provided should be relevant and useful;• evaluation outputs are delivered at appropriate and relevant times in apartnership’s development;• results are made widely available to a range of stakeholders and presented inappropriate, accessible formats;• recommendations should be realistic in terms of the ability of the partnershipto respond.Another evaluator stressed the importance of engendering a relationship with thepartnership in which the partnership board and programme team recognised thevalue of evaluation and were open to both positive and more critical feedback:I would say that the management team were very open to our approach toevaluation which is to identify the positives and to be honest about what couldbe done better… one of the things that I think made the delivery of theevaluation effective for us was that we shared the same values as theprogrammeProblems using evaluation in a context of changeMany local evaluation reports and evaluators interviewed, however, emphasised anumber of factors that had inhibited their ability to effectively influence partnerships’decision making. These stem from the climate of ongoing change nationally andwithin local authority areas, meaning that, in practice, local evaluation of theChildren’s Fund has tended to be politically/tactically driven (drawing on Young’s2002 framework).The Children’s Fund initiative presents a number of challenges in seeking to applylocal evaluation findings to decision making and practice. The rapidly changingnational context means that changes in partnerships’ directions are often politicallyChapter 3 41

motivated rather that based on local evaluation. The introduction of the CentralGovernment stipulation in 2002 that 25% of programmes’ budgets are allocated tocrime prevention activities (CYPU and the Youth Justice Board, 2002), for example,created significant tensions between Central Government priorities and the abilitiesof partnerships to determine their own programmes (Morris & Spicer 2003; Mason, etal., 2005).Partnerships also experience considerable change locally as mainstream health andsocial care agencies embrace a number of alternative and sometimes divergingagendas. Reflecting this context, as the ongoing work of the National Evaluationsuggests, Children’s Fund partnerships have sometimes struggled to establish theiragendas (for example, NECF, 2004b & c). Indeed, a small number of local Children’sFund evaluators indicated that evaluation was to an extent being driven by shiftingmainstreaming agendas in which partnerships are under pressure to use evaluationto both demonstrate what Children’s Fund projects are achieving to mainstreamagencies and to demonstrate the effectiveness of new practices promoted by theinitiative. The implications of this were described by a Wave One local evaluator:… they had their own agenda and that was to do with the migration intochildren’s trust and the emergence of integrated services; I just saw thatagenda kind of looming large… I think by the end of my time of evaluation withthe Children’s Fund I had a sense that the Children’s Fund were really comingup against that limit… things becoming much more top down and much lessbottom up. It was almost as though the Children’s Fund had done its job; it hadcreated some good and interesting work… and some projects were being kindof cherry picked for mainstreaming; the others were simply being let go to thewall.Some evaluators indicated that they experienced difficulties controlling howevaluation outputs would be disseminated, which groups would receive themessages, and how outputs would be used. The evaluator of a Wave Onepartnership highlighted the ways the messages contained within evaluation outputshad to be carefully poised in terms of reporting on negative aspects of partnerships’work; indeed, senior figures within the partnership had sought to exercise controlover the distribution of evaluation material:… it’s a little bit sad that it’s so politically driven now, and it is, and reports havebeen written very, very sensitively. I don’t know whether you noticed that, butthey have been written very sensitively. And we have been a bit pressuredabout where the reports go and… well I haven’t but powers above have, wherethey go, and if they’re in the public domain.42Chapter 3

Similarly an evaluator of a Wave Two partnership suggested:…because it’s almost like they know what they want to find, and they’re justusing the evaluation to get there. But they know exactly what they want to find;you know almost the conclusions. I could work the conclusion now, and I’veonly just started the evaluation, I’m only on week one of my phase threeevaluation.A related particular problem inhibiting partnerships’ abilities to absorb and act onevaluation material, noted by a local evaluator appears to stem from strategicstakeholders’ multiple priorities and their commitments to their own agencies andother partnerships in the locality meaning they tend to have insufficient time to act onevaluation findings in a climate of rapid change:… what they’ve said is, ‘we’d like to get more involved, but… we’ve got our dayjobs. This is just one partnership board that we’re on, we’re on three, we’re onten, we’re on eight. So we can’t input into it as much as we can’… So in termsof their learning, I think it’s quite minimal. But then having said that, because ofall the other things that are going on in the borough, that doesn’t mean that theydon’t know about them, or indeed that they don’t want to know about them.Evaluators’ accounts suggest, however, that partnerships may be able to respondmore fully to evaluation findings at different stages in their development. According toan evaluator of a Wave Two Children’s Fund partnership, the partnership was opento receiving lessons from the evaluation and act on them in its early stages: I thinkwhat also helped the evaluation was that the programme was at a phase where itwas open to suggestion. So it was kind of malleable, so if I said, we really need amore consistent referral process, or we really need the monitoring stats to beanalysed, it was being done. The same local evaluator noted how the partnershipbecame less open to criticism during the course of the evaluation as the localauthority began to move towards the integration of children’s services and children’strusts. Whereas at the inception of the evaluation the partnership was very open tochange, as the evaluation has progressed the resulting reports have to be writtenvery, very sensitively…we have been a bit pressured about where the reports goand…if they’re in the public domain.Perceptions of the value and credibility of local evaluationA further key issue influencing evaluators’ abilities to influence the decision making ofChildren’s Fund partnerships is that, in some cases, local evaluators experienced adegree of dissonance between their views on what represented legitimate evaluationmethods, and those of some partnership board members. As noted earlier, theChapter 3 43

legitimacy of qualitative methods and multiple (quantitative and qualitative) evaluationmethods is being increasingly recognised among UK policy makers as an appropriateapproach to evaluating complex interventions (Coote, et al., 2004; Spencer, et al.,2004). Indeed, as outlined in Chapter 2 of this report, local evaluators have tended tofully embrace multiple methods, and hence recognise that different methods aremore appropriate in examining different elements of a programme’s work. Anevaluator from a Wave One partnership, for example, described how the partnershipboard tended to favour ‘hard’ quantitative evidence rather than qualitative evaluationdata, and found it difficult to accept the legitimacy and relevance of the latter:It might just be some kind of headline statistics they want really…I know thatthe drop-in [project] is a life-line to some of those parents that attend there, theywouldn’t know what to do with their children if they didn’t go to that drop-in, andI can write that up but it’s not hard evidence, just the stats really whichsometimes doesn’t really tell the whole story… And even though we might havesay all the stats they’re wanting, what the actual people on the ground mightsay might be something completely different.Similarly, a Wave Two evaluation described how partnership board members’ limitedunderstandings of notions such as indicators and impacts had been a frustratingbarrier to progress in terms of their unrealistic expectations of the evaluation:The partnership board didn’t really understand... why can’t you show how gooda service is?... And I said, well actually, if you haven’t got those indicators inplace, I can’t magic them out of the air. And you need to have them on anongoing basis, because I can’t do them in retrospect, because you should havebeen recording what risk factor the children came in with, effectively what’sbeen done with them and what they came out with. I can do what they cameout with, but I can’t compare that to anything at that point of entry.Many of the reports also remained tentative in offering definitive findings on progress,and have given observations that can continue to inform an evolving programme thatis formative. This has been because local evaluators have had to take account oftime lapses in the expected progress of programmes resulting from problems ofgetting programmes started in some parts of the country, and funding uncertaintiesencountered by programmes that have slowed progress. Additionally, localevaluators have produced final reports when some programmes are about half waythrough their lives resulting in evaluations being short or medium term in duration. Asa consequence, local evaluators have been reluctant to identify lasting outcomes.Hence, there can be a tension between providing evaluation outputs that areappropriate for readers in terms of being short and using accessible terms asopposed to fully evidencing statements made and fully exploring complex issues.44Chapter 3

Evaluators sometimes struggled to address the multiple requirements of theevaluation with the timing of their engagement and the expectations of when theywould feedback. It was recognised, for example, that early engagement andfeedback might assist programmes to plan their development on an informed basis.However, this would limit that evaluation’s ability to present what were seen asdefinitive conclusions. Conversely, later evaluation engagement/feedback were seenas providing a greater degree of confidence in findings, but as a retrospective viewthat may be less helpful in informing a partnership about future directions.Other evaluators pointed to the importance of maintaining ongoing dialogue withprogramme managers and members of the programme team in order to ensure thatevaluation outputs are useful, accessibly presented and that they are realistic aboutwhat the evaluation is able to produce. For example, one evaluator pointed to theproblems of not effectively managing the expectations of the partnership:… the previous director didn’t do enough in the way of dialogue with us…whathappened was that we were producing types of report and working on focuseswhich he didn’t particularly want or didn’t find particularly useful…whathappened in the end though was that the… clearly regretted the fact that he’dcome in half way through and hadn’t had a hands on approach because he feltvery strongly that for local evaluation to work it had to be a sort of collaborativething and in fact at times I feel he wants to be slightly too hands on for it to becalled evaluation to be honest.Another evaluator described a number of problems that limited the impact that theevaluation had on the work of the partnership. Such problems included limited clarityamong stakeholders about the purposes of evaluation, that the local evaluation hadproduced outputs that were too ‘academic’ in style and that it had not been able toproduce ‘hard’ data demonstrating outcomes which the programme required:… we began to get this sort of critical feedback about how you know the wholething was too academic and too weighty and it wasn’t telling them what theyneeded to know and so on and when we tried to get to the bottom of this werealised that they were expecting to walk away with sets of targets met and youknow hard data about outcomes and things; which we were never being askedto find actually.3.4 Chapter Summary1. Evaluators have each adopted a range of summative, formative, and forsome, dialogic approaches to engaging with partnerships reflecting theirChapter 3 45

multiple roles at different stages in partnerships’ development. Evaluationmaterial is disseminated through reports, presentations and workshops,websites, and some evaluators attend partnership board meetings, subgroupsor other organisational structures. A number of evaluators emphasised theimportance of ongoing dialogue with strategic partners about theirdevelopment rather than simply providing summative evaluation outputs.2. The importance of adopting appropriate means of feeding back evaluationmessages to a range of stakeholders including strategic partners, serviceproviders and children, young people and their families who used Children’sFund services was recognised by some evaluators. Some produced ‘childfriendly’ versions of reports or disseminated findings at events that involvedchildren and young people. However, it would appear that few evaluatorsprioritised service users, including children and young people, as an importantaudience of evaluation findings.3. Many local evaluations developed approaches to raising the capacity ofChildren’s Fund projects to self-evaluate their work through means such asworking alongside project staff in developing appropriate self-evaluation‘tools’, or through providing training.4. A number of local evaluation reports identified where their recommendationshave been developed into tangible changes in the partnership. Others wereuncertain about how evaluation material had or would influence thepartnership. Evaluations appear to be offering material to partnerships to beused instrumentally, that is, through changing their practices and/or decidingwhich projects to re-commission. Local evaluation also appears to be usedconceptually in terms of expanding partnerships’ understandings relating toprevention, partnership working and participation (that is an enlightenmentmodel of the relationship between evaluation and decision making).5. Evaluators suggested that they could have more influence on partnerships byproviding findings that are relevant, useful, timely, accessible to a range ofstakeholders, and provide realistic recommendations. The importance ofengendering a relationship in which the partnership recognises the value ofevaluation and is open to both positive and more critical feedback was alsonoted.46Chapter 3

6. A number of factors inhibit evaluators’ influence on partnerships’ decisionmaking. Pressures from ongoing change nationally and within local authorityareas as mainstream agencies embrace a number of alternative, andsometimes, diverging agendas means that local Children’s Fund evaluationmay get drawn on selectively to legitimise decisions or to demonstrate thesuccesses of partnerships’ work (that is a politically/tactically driven model ofthe relationship between evaluation and decision making).7. Some evaluators indicated that they experienced difficulties controlling thedissemination of evaluation outputs, how outputs would be used, and indeedhad experienced difficulties reporting on negative aspects of partnerships’work. In some cases Children’s Fund board members’ multiple priorities andtheir commitments to their own agencies and other partnerships meant theyhad insufficient time to act on evaluation findings.8. A number of evaluators experienced disagreement with stakeholders aboutthe legitimacy of different evaluation methods. Some stakeholders favoured‘hard’ quantitative evidence and rejected the value of qualitative methods,whilst other appeared to have limited understandings of notions such asindicators and impacts or had unrealistic expectations of what evaluationscould produce. Maintaining ongoing dialogue in order to manage partnerships’expectations is therefore important.Chapter 3 47

Chapter 4: Prevention, Participation and PartnershipWorking: Key Messages from Children’s FundLocal Evaluation Reports.This chapter reflects on a number of key messages presented within local evaluationreports and how these relate to the themes driving the Children’s Fund programme,namely, prevention and preventative services, children and young people’sparticipation and partnership working. Some of the ways local evaluators havesought to conceptualise prevention, participation and partnership working are alsohighlighted.4.1 Key Messages from Local Evaluation: Prevention and Preventative ServicesThe focus of local evaluators’ work on prevention has been to analyse the activitiesof Children’s Fund projects. Relatively few reports concentrate on the place ofprevention and early intervention, other than in very broad policy terms, such as howthe local prevention strategy relates to Children’s Fund planning. Hence, mostevaluations have sought to conceptualise prevention in relation to the Children’sFund sub-objectives, and/or to the outcomes framework in Every Child Matters intheir assessments of projects’ progress. A number of evaluators also drew on thefour ‘tiers’ of prevention (diversionary; early prevention; heavy end prevention; andrestorative prevention) adopted in the Children’s Fund Guidance based on the workof Hardiker, et al. (1991) as a frame of reference for locating the activities ofChildren’s Fund services.As noted above, measuring/demonstrating effective prevention was most widelyattempted through quantitative analysis of project monitoring/secondary statisticaldata sets and multiple methods in which service providers and service users’perspectives were elicited in order to attempt to show that preventative services hadhad a positive impact on service users’ circumstances and to identify the practicesthey employed that were effective. There were many positive messages in localevaluation reports in relation to the beneficial impacts of Children’s Fund preventiveprojects. Illustrating the sorts of conclusions evaluations are reaching on preventativeactivities a report on a Wave One partnership summarises that:The children and young people who access services funded by the X Children’sFund are benefiting from the support that is being provided. The Fund isreaching and supporting children and young people who are suffering socialexclusion, some of them at high levels. Families are also benefiting from thework of the X Children’s Fund in ways that are helping to reduce the effects ofpoverty… The X Children’s Fund has actively supported innovative work whichpoints to new models of working with children and young people.Chapter 4 49

A number of evaluations described how partnerships had effectively introducedprojects focusing on early intervention and drawing in traditionally ‘hard to reach’groups. For example, a Wave One evaluation report suggests: ‘…a large number [ofprojects] have focused on reaching commonly excluded groups and in supportingthem to access mainstream services’. Similarly, a report of another Wave Onepartnership states that: ‘…some projects were using the Children’s Fund tospecifically target children often excluded from mainstream provision. These includedprojects whose primary focus was raising awareness in order to foster understandingand integration’.Some evaluators also described how drawing upon the skills and expertise of thevoluntary and community sector has been a particularly important factor in effectivelyengaging and addressing the needs of traditionally excluded groups. For example, aWave One evaluation report describes how voluntary and community organisationshave been able to offer alternative approaches to those of the mainstream:… families and children are prepared to work with Children’s Fund services thatare non-stigmatising, voluntary and non-compulsory, flexible and accessible. Asa result services have developed a variety of methodologies and approachesthat meet the needs of children and young people and in some instances theirfamilies.Similarly, a Wave One evaluator suggested that voluntary/community-led Children’sFund projects were able to provide more flexible services than the statutory sector:…there is some evidence that voluntary or community sector led projects tend to bemore flexible in the timing or ‘coverage’ of the services they offer when compared tostatutory led projects (e.g. 50% vs. 18% were accessible at weekends).Other evaluations describe how developing collaborative arrangements with otherlocal initiatives and/or agencies has been important to the success of Children’s Fundpreventive services. One evaluation report, for example, describes how thecollaboration between Children’s Fund and other local preventive strategies such asSure Start and the Teenage Pregnancy Strategy gave impetus to the work ofChildren’s Fund funded outreach workers who have successfully offered support tothose families who are not normally in contact with any formal services. A furtherexample of this is given in a Wave One evaluation report that describes a familysupport service which is successfully ‘fostering closer working between agencies in50Chapter 4

order to provide effective preventive services for children and families’. Thepartnership supplies workers who provide support in homes and schools in responseto children and parents’ needs. The project is described as successfully focussing onlow level preventive work by supporting families ‘… who are struggling and have aneed but who are not yet at crisis intervention point and whom may be distrustful ofsocial services or other professionals’.A key message noted in local evaluation reports is that there tend to be divergentunderstandings of ‘prevention’ among stakeholders within partnerships, althoughprogress is being made by some partnership boards in negotiating a consensus ofunderstanding. This corresponds with the findings of the work of the NECF (NECF,2004d). One local evaluation report, for example, notes that within the partnership:‘There is no clear single agreed unitary definition of the meaning of prevention’.Similarly, another local evaluation suggested:If there is no shared understanding [from interviews with key stakeholders] as towhat prevention means there is a risk of confusion in implementation. Indeed asthese extracts demonstrate prevention means so many different things, itperhaps risks meaning nothing.Some reports highlighted that some service providers tend not to explicate what theyhope to achieve and how their activities are connected with prevention. Indeed, anumber of evaluators noted very divergent views among service providers about theiraims, for example a report on a Wave One partnership states:Some project managers said that they provided a service to ‘plug the gap’ anddid not think it was necessary to change any practices whilst others had moreof a reform agenda with clear explanations of how their services provided newopportunities to make a difference to the everyday lives of disadvantagedchildren and young people.It was also noted in a number of reports, however, that despite limited clarity aroundprevention in terms of what this means and how this can be applied, a number ofpartnerships were linked to networks and partnerships that were emerging locallythat were taking thinking on prevention further. A local evaluation report from a WaveOne partnership, for example, describes how a youth crime prevention project hascreated a multi-disciplinary panel to decide on preventive interventions for youngpeople at risk of offending. As the panel consisted of key strategic players in the localauthority, it is not unlikely that such discussions around prevention have been heardat partnership board level.Chapter 4 51

4.2 Key Messages from Local Evaluation: Children and Young People’sParticipationLocal evaluators aimed to establish Children’s Fund stakeholders’ understandings ofthe aims of children and young people’s participation. Within some programmespartnership boards appear to have agreed understandings of participation; thesetend to equate to a ‘consult and inform’ model of participation whereby children’sviews and experiences are elicited in order to inform the design and development ofchild focused and responsive services. In most programmes, however, localevaluators reported diverse views of how participation should be defined and appliedranging from promoting children’s involvement in all aspects of service design anddevelopment, through to some circumspection among stakeholders about thedifficulties of overcoming current ways of working whilst avoiding tokenistic gesturesof participation. Nevertheless, evaluators reported that professionals and adults’definitions of participation have tended to predominate, rather than those of childrenand young people themselves. Similarly, among projects evaluated, participation wasprimarily equated with consultation activity. Recommendations made in localevaluation reports emphasise the need for partnerships to develop shared definitionsof participation and to develop a coherent strategy in implementing participation.Such findings correspond with those of the NECF (see NECF, 2004c).Within a number of evaluation reports participation is conceptualised in relation tohierarchical typologies. Different activities are depicted in terms of the extent to whichchildren initiate or control the decision making process as a framework for locatingChildren’s Fund activities. Many local evaluators described high degrees ofcommitment to participation among stakeholders. However, in practice, progress inintroducing participative activities tended to be reported as relatively limited andactivities often correspond with lower degrees of initiation or control according to thehierarchical scales.Many local evaluators pointed out, particularly in interim reports in which they tendedto comment on the early stages of Children’s Fund partnerships’ development, thatpartnerships were under pressure to prioritise setting up their programmes over andabove establishing the principles of how to generate meaningful engagement withchildren, parents and indeed the wider community. Some evaluators accepted,however, that they had offered preliminary findings that were based on evidence ofemerging practices rather than being able to comment on the longer term effects ofpartnerships’ work on participation.52Chapter 4

Nevertheless, participation in practice typically excluded children, young people andtheir parents/carers from direct engagement in decision making leading to thedevelopment of the partnership strategically or in particular projects. However, someevaluators pointed to examples of such forms of participation, albeit inconsistentlyapplied across partnerships’ activities. Where participation was reported it wasusually where children and young people had been respondents to consultationexercises rather than active involvement in decision making relating to servicedevelopment. An evaluation report for a Wave One partnership, for example,suggested that despite broad levels of commitment, the practice of introducingparticipation that was meaningful to children and young people was more limited:There was evidence of a commitment to participation through key objectivesetting, resources being dedicated and the monitoring of the levels and types ofparticipation. From children’s interviews there was limited evidence of thenumbers of children involved in service planning and development, withchildren saying they would welcome the opportunity of further involvement.A number of local evaluators pointed to Children’s Fund partnerships and projects’difficulties in ensuring that children and young people participating wererepresentative of the range of characteristics of children locally (a finding that is alsonoted in NECF, 2004c). Children and young people who were assumed to be ‘hard toreach’, such as those from refugee families and young carers, were seen asparticularly under-represented in participative activities. There are, however,exceptions, for example, a Wave One local evaluation report describes how a projectsupporting black and minority ethnic and refugee communities was encouragingactive participation in the development of the service by responding to their ‘selfidentifiedneeds’. This project was described as succeeding in its purpose of being‘empowering’, by developing ‘a variety of methodologies and approaches that meetthe needs of children and young people’. Evaluators also commented on the ageprofile of children who had participated in projects as being older, and that stepsneeded to be taken to ensure younger children’s views were also being heard. Someevaluators commented on the need to ensure that inappropriate generalisations werenot made from a small group of children, and that children were not being calledupon to represent large constituencies of views when no arrangements were in placefor the range of views and ideas to be captured.Local evaluators identified key processes that a number of partnerships had adoptedthat had appeared to have effectively promoted children and young people’sparticipation. Among these is the importance of resourcing dedicated skills and timeChapter 4 53

for staff, typically dedicated participation officers, to work on promoting participativestrategies, the need to train practitioners and service managers and to provideorganisational support for Children’s Fund service providers to develop effectiveapproaches to participation. It is also widely acknowledged that considerableinvestment of time is likely to be required in order for participation strategies tobecome effective, hence many evaluators recommend that partnerships factor inrealistic timescales in enabling participative practices to become an integral part ofservice delivery and strategic involvement.The positioning of voluntary and community sector organisations within partnershipswas identified by a number of local evaluators as being an important factor inencouraging a wide range of stakeholders to participate in Children’s Fund activities,particularly at operational level. A Wave One report, for example, suggests thatvoluntary and community organisations have the skills to draw ‘…families andchildren who have not traditionally engaged well with mainstream services [intoparticipative activities]’. Several reports describe how voluntary and communityorganisations have been able to bring a more flexible approach to project activity,and are more willing to experiment with appropriate and innovative ways ofaccessing and involving service-users.Some reports raise the concern, however, that lessons relating to participatory workcould potentially be lost if engagement with the voluntary and community sector isnot sustained. This could be done, for example, by ensuring that resources are inplace to support ongoing staff training. Similar concerns related to sustainability weredescribed in other evaluation reports; a number of evaluators highlighted theimportance of partnerships having in place strategies, protocols and guidance tobring coherence to participative practices. For example, a Wave One local evaluationasserts that in order to ensure ongoing accountability, permanent structuressupporting participatory practices need to be implemented and sustained:‘Partnerships need to be both imaginative and transparent, with clarity of objectivesand processes’.4.3 Key Messages from Local Evaluation: Partnership WorkingMany reports analysing partnership working adopted the Guidance document as areference point in defining expectations of how partnership boards should beoperating. Some later reports also referred to the emerging policy context ofpartnership working described in Every Child Matters to help define important54Chapter 4

dimensions to partnership working. Early interim reports concentrated on describingthe activities of partnership boards in setting up their programmes of work. Laterreports focused on the wider aspects of partnership working in terms of the way itrelates to the changing policy context towards integrated children’s services followinglegislative changes, placing more emphasis on mainstreaming and outcome analysis,with many making recommendations to widen the membership of the partnershipboard. For example, an evaluation report on a Wave One partnership points to theimportance of sustaining partnership working as new configurations of children’sservices emerge. It is suggested that information-sharing and networking betweenuser groups, policy makers and practitioners ‘…can provide a strategic basis formainstreaming Children’s Fund work’.A number of evaluation reports noted that Children’s Fund partnerships tended tostart with small groups of statutory sector senior managers who assembled theinfrastructure of the local programmes and commissioned services. Within somepartnerships this was followed by a gradual expansion in the size and makeup ofstrategic stakeholders, although comments from stakeholders suggest that numbersof board members have declined over time and that more junior managers werereplacing some original senior managers. This had both eroded the continuity ofstakeholders’ contributions as well as reflecting a lowering of the priority of theChildren’s Fund by some agencies. Local evaluators also noted that few partnershipboards had regular health service representation at meetings.Local evaluators reported different experiences and understandings of partnershipworking among partnership board members. They also found that discussions hadtaken place and broad agreements reached on what partnership working meant interms of governance arrangements, the vision or direction of the partnership and acommitment to the structures and processes to take the partnership forward. Localevaluators noted that some members saw the Fund in administrative terms with theresponsibilities of the board being to ensure probity and demonstrating that it metCentral Government requirements. In contrast some board members viewed theirpartnership as an expression of close collaborative working between agencies andsectors to help bring about cultural change in the way children’s services weredelivered. In particular, partnership working was viewed as providing opportunities forsharing information and learning about good practice as well as agreeing on servicedevelopment and shared priorities and thereby reducing duplication of services.Chapter 4 55

Where partnership working was described as being effective, evaluators describedthe conditions that promoted that effectiveness and recommended preserving thesefeatures while being able to effectively adapt to a changing policy environment andthe financial challenges this implied. Important conditions include having a sharedvision of the purpose of the initiative and transparency of decision making where itwas clear to stakeholders where responsibilities lay, what decisions were being madeand what actions were being taken by whom. A critical condition for effectivepartnership working was noted as good communications between stakeholdersrepresenting the different aspects of a programme including the partnership board,Children’s Fund staff, service providers and service users. This was particularlynotable in terms of perceived relationships between the operational and strategiclevels of the Children’s Fund partnership, the lack of which was regarded as a majorbarrier to the ‘mainstreaming’ of Children’s Fund practices and philosophy. Thisdifficulty was sometimes exacerbated by what is described by a Wave Two localevaluation report as ‘…complicated linkages between the vertical and horizontallayers of the partnership’s structure’.Some local evaluations identified cases of relatively adversarial relations betweenboard members, in which unresolved differences existed between board members,sometimes stemming from negative experiences of partnership working betweenagencies prior to the Children’s Fund. This was described as having an adverseimpact on the effectiveness of the partnership board, producing fragmented, singleagency and unfocused board activities.Local evaluators also described the collaborative arrangements between Children’sFund projects, noting however, that such arrangements tended to be relativelyuneven across partnerships’ activities. For example, an evaluation report of a WaveOne partnership suggested: ‘There were piecemeal operational relationships acrossprojects that were forged through some enthusiastic staff developing contacts’.Nevertheless, some reports refer to established and emerging networks of voluntaryand community sector organisations that provide Children’s Fund services. Whilstsome networks were facilitated by Children’s Fund staff in order to share informationand good practice, many were reported as having been established by theorganisations themselves.A number of local evaluators commented on the relationships between statutory andvoluntary and community organisations in Children’s Fund partnership arrangements56Chapter 4

and that in many localities, local authority agencies took the lead in programme setup, whilst voluntary and community organisations tended to be more active later intheir roles as service providers. Some reports note a degree of frustration amongvoluntary and community stakeholders who felt they had relatively limited influencecompared to some statutory agencies, although this was not the case across allpartnerships. In some partnerships voluntary and community organisations had ahigh profile in partnership board activities and the Children’s Fund was seen as bothraising the profile of the voluntary and community sector and as promoting productiverelations with statutory sector agencies.A number of more recent evaluation reports speculated on the ways the relationshipsbetween the statutory and voluntary and community sectors may play out aschildren’s trusts become established. A commonly-held anxiety is that gains madethrough Children’s Fund activity may be lost in the next phase of integratingchildren’s services as it was predicted that the voluntary and community sector wouldbe relatively marginal in decision making. Illustrating this point one report suggests:The Children’s Fund has assisted working relations between the statutory andvoluntary and community sectors, but some voluntary organisations seethemselves as being at risk of again being on the margins with the move to thelocal authority taking the lead in integrating children’s services through theChildren’s Trust.However, some local evaluation reports point to concerns among voluntary andcommunity sector stakeholders as closer relationships between themselves and thestatutory sector might undermine the distinctive skills and working practices that havebeen deemed valuable within Children’s Fund partnerships. For example, a WaveOne evaluation described how the ‘anti-managerial’ culture of voluntary andcommunity organisations and their independence had facilitated work with ‘hard-toreach’families: ‘… sometimes independence becomes necessary for voluntary andcommunity organisations to maintain a critical distance from mainstreamarrangements for the benefit of particular groups of children’.Some local evaluators identified links between partnership boards and other strategicgroups in the locality, and that board members were familiar with policy changes inother sectors that would impact on the Children’s Fund programme. Other reportsrecorded mixed experiences among strategic Children’s Fund stakeholders abouthow well connected they were to other strategic groups, with some board membersacross different programmes tending to feel relatively uninformed about widerChapter 4 57

strategic changes in children’s services. Some evaluators noted that partnershipboards generally felt dislocated from the broader strategic environment, andrecognised they needed to connect with other strategic groups to prepare for themainstreaming issues concerning the future of the Children’s Fund. Some localevaluators also noted, however, that emerging networks and working practices wouldbe valuable as partnerships prepared for the move towards more integratedchildren’s services, as one Wave Two report suggests:… [there has been] considerable progress in breaking down professionalbarriers between respective organisations that traditionally have not workedeffectively together. This would appear to be one of the key legacies of the XChildren’s Fund that needs to be sustained in the Children’s Trust.4.4 Chapter Summary1. The focus of local evaluators’ work on prevention has been to analyse theactivities of Children’s Fund projects. Most evaluations conceptualisedprevention in relation to the Children’s Fund sub-objectives, and/or to theEvery Child Matters outcomes framework in assessing projects’ progress. Anumber of evaluators related the work of projects to the four ‘tiers’ ofprevention adopted in the Children’s Fund Guidance.2. Local evaluators widely report on the beneficial impacts of Children’s Fundpreventive projects on children and young people’s lives and that manyprojects are effective at targeting ‘hard to reach’ groups of children and youngpeople. Some evaluators suggest that voluntary and community organisationshave been particularly successful in engaging and addressing the needs oftraditionally excluded groups through offering alternative approaches to thoseof the mainstream agencies in terms of being flexible, accessible and nonstigmatising.3. Understandings of ‘prevention’ among strategic stakeholders and serviceproviders within partnerships tend to be described in local evaluation reportsas divergent, although some acknowledge that progress is being made bypartnership boards in negotiating a consensus of understanding.58Chapter 4

4. Evaluators identified diverse definitions of participation among stakeholders inmost partnerships. Many evaluators reported that professionals’ and adults’definitions of participation tend to predominate, rather than those of childrenand young people themselves. Often these equate to consulting children andyoung people in order to inform the development of responsive services.Despite considerable commitment to participation, in practice progress tendsto be relatively limited, with few examples of children and young people’sdirect participation in decision making at strategic and project levels.5. A number of evaluators suggested that ‘hard to reach’ children and youngpeople and younger children were particularly under-represented in Children’sFund participative activities, although some evaluators identified examples ofparticipative activities that had been effective in engaging an inclusive rangeof children and young people.6. Local evaluators pointed to the importance of partnerships resourcingdedicated participation officers, the need to train practitioners and servicemanagers and the importance of providing organisational support for serviceproviders to develop effective participative practices. Considerable investmentof time is also needed if participation strategies are to be effective. Someevaluations suggest that voluntary and community organisations may beparticularly effective in promoting participation due to their abilities to work inflexible and innovative ways.7. Evaluators reported different experiences and understandings of partnershipworking, although within many partnerships agreements had been reached onwhat partnership working means. Some stakeholders saw partnership boards’responsibilities as ensuring probity and demonstrating that they met CentralGovernment requirements. Others emphasised collaborative working betweenorganisations to bring about cultural change in children’s services.8. Conditions reinforcing effective partnership working include: sharing a visionof the purpose of the initiative; transparency of decision making; clarity aboutstakeholders’ responsibilities; good communications between groups ofstakeholders including members of the partnership board, Children’s Fundstaff, service providers and service users.Chapter 4 59

9. Evaluators reported that some collaborative arrangements and networksbetween Children’s Fund projects had been facilitated by partnerships,although many suggested that such arrangements are relatively unevenacross partnerships’ activities. Other collaborative arrangements and networkswere reported as having been established by the organisations themselves.10. A number of evaluators commented that statutory agencies tended to leadstrategically, whilst voluntary and community organisations tended to provideChildren’s Fund services. However, within some partnerships voluntary andcommunity organisations had been strategically influential, indeed, theChildren’s Fund had raised the profile of the sector. Some stakeholderspredicted, however, that the voluntary and community sector would bemarginalised in the next phase of integrating children’s services.60Chapter 4

ConclusionsLocal evaluators have responded to the challenges of evaluating the complex, multilayeredarrangements developed by Children’s Fund partnerships by creativelyadopting evaluation methods. Many have drawn on multiple quantitative andqualitative methods, as well as data sets at a number of different levels, to show theimpacts of projects and other activities, or used methods differentially to answerquestions about different aspects of partnerships’ work. A widely experienceddilemma, however, relates to the difficulties in definitively demonstrating the impactsattributable to specific projects or activities, reflecting the complexity of the initiativeand limited time frames within which evaluations have worked. Some evaluators arevery forthright in acknowledging the limits to what they are able to show, whilst othersacknowledge the importance of openness and transparency in the evaluationprocess and the methods used in terms of data collection, analysis andinterpretation. What this highlights is the importance of evaluators managingstakeholders’ expectations about what evaluation is able to show, and its limits.Many evaluators also recognise the value of evaluation evidence based on serviceusers, and in particular, children and young people’s voices, although the extent towhich children and young people have been engaged in evaluation activities is veryvaried. The idea that children’s voices should be heard, valued and acted upon isconsistent with the participative ethos of the Children’s Fund initiative. However, anumber of factors have made it difficult for evaluators to work in this way. It is widelyrecognised that additional time and resources may be required to effectively workwith children and young people. Moreover, many evaluators describe a degree ofcircumspection, and occasionally hostility to qualitative methods such as interviewingin which children and young people express their views about their use of projectsand their perceptions of the benefits of using them. Such methods are frequentlyseen as more ‘anecdotal’, with the implication that they produce lesser forms ofevidence than ‘hard’, quantitative outcome. Nevertheless, the value of qualitativemethods in social policy research is increasingly recognised in governmentdepartments. This highlights the need for evaluators to carefully work withstakeholders to develop clear understandings of the ways difficult sets of evaluationmethods can be effectively used in combination to assess the work of differentelements of programmes’ work.Conclusions 61

The importance of providing findings that are relevant, useful, timely and accessibleto a range of stakeholders appears to be widely recognised by local evaluators, as isthe importance of providing realistic recommendations to partnerships. Many localevaluators regarded ongoing dialogue with partnerships as critical to successfullyinfluencing their work through engagement in partnership structures such asevaluation sub-groups or disseminating material through events, rather than simplywriting reports for strategic audiences. Indeed, running events may be a moreinclusive way of disseminating their findings to strategic stakeholders as well as frontlineservice providers and service users including children and young people.However, it is evident that many local evaluators have, in practice, experienceddifficulties influencing the development of partnerships’ strategies and practices, orare unclear about how their findings have been translated into strategies andpractice. Some evaluators suggested they had experienced difficulties maintainingtheir independence, in reporting on negative aspects of partnerships’ work or thatpartnerships had drawn on evaluation material selectively to demonstrate thesuccessful aspects of their work or to legitimise decisions. This was seen in thecontext of pressures from ongoing change nationally and within local authority areas.Partnership board members’ multiple priorities and their commitments to their ownagencies and other partnerships meant they were unable to act on evaluationfindings or had insufficient time to do so. This highlights the challenge that localevaluators have had in providing material that is relevant and useful to partnershipswhilst having the space to produce balanced, independent accounts of thepartnerships’ work.62Conclusions

ReferencesBecker, S. and Bryman, A. (eds) (2004) Understanding Research for Social Policyand Practice: Themes, Methods and Approaches, Bristol: The Policy Press.Clarke, A. (2001) ‘Research and the policy-making process’ in N. GilbertResearching Social Life (2 nd ed.), London: Sage.Coote, A., Allen, J., and Woodhead, D. (2004) Finding Out What Works:Understanding Complex, Community-Based Initiative, London: Kings Fund.Coulton, C., Connell, J., Kubisch, A., Schorr, L., and Weiss, C. (eds) (1995) NewApproaches to Evaluating Community Initiatives: Concept, Methods and Contexts,New York: The Aspen Institute.Children and Young People’s Unit (CYPU, 2001) Children’s Fund Guidance, London:DfES.CYPU and Youth Justice Board (2002) Use of Children’s Fund Partnership Fundingfor Crime Prevention Activities Jointly Agreed with Youth Offending Teams:Guidance.Davies, H., Nutley, S., and Smith, P. (eds) (2000) What Works? Evidence-BasedPolicy and Practice in Public Services, Bristol: The Policy Press.Donaldson, S. & Scriven, M. (eds) (2003) Evaluating Social Programmes andProblems: Visions for the New Millennium, London: Lawrence Erlbaum Associates.Edwards, A., and Fox, C. (2005) ‘Using Activity Theory to evaluate a complexresponse to social exclusion’, Education & Child Psychology 21:1, pp50-60.Hardiker, P., Exton, K., and Barker, M. (1991) Policies and Practices in PreventiveChild Care, Aldershot: Ashgate.Mason, P. Morris, K., and Smith, P. (2005) ‘A complex solution to a complicatedproblem? Early messages from the national evaluation of the Children’s Fundpreventative programme'. Children & Society 19:2, pp131-143.McDonald, G. (2000) ‘Social Care: rhetoric and reality’ in Davies, H., Nutley, S., andSmith, P. (eds) What Works? Evidence-Based Policy and Practice in Public Services,Bristol: The Policy Press.Morris, K., and Spicer, N. (2003) The National Evaluation of the Children’s Fund:Early Messages for Developing Practice, London: DfES.National Evaluation of the Children’s Fund (NECF, 2004a) Assessing the Impact ofthe Children’s Fund: The Use of Indicators, Birmingham: University of Birmingham.National Evaluation of the Children’s Fund (NECF, 2004b) Collaborating for theSocial Inclusion of Children and Young People, London: Department for Educationand Skills.National Evaluation of the Children’s Fund (NECF, 2004c) Children, Young People,Parents and Carers’ Participation in Children’s Fund Case Study Partnerships,London: Department for Education and Skills.References 63

National Evaluation of the Children’s Fund (NECF, 2004d) Prevention and EarlyIntervention in the Social Inclusion of Children and Young People, London:Department for Education and Skills.Percy-Smith, J. (2000) Policy Responses to Social Exclusion: Towards Inclusion?Maidenhead: Open University Press.Pierson, J. (2002) Tackling Social Exclusion, London: Routledge.Solesbury, I. (2001) Evidence Based Policy: Whence it Came and Where it is Going,UK Centre for Evidence Based Policy and Practice Working Paper 1, London: ESRC.Spencer, L., Ritchie, J., Lewis, J., and Dillon, L. (2004) Quality in QualitativeEvaluation: A Framework for Assessing Research Evidence, London: Cabinet Office.Young, K., Ashby, D., Boaz, A., and Grayson, L. (2002) ‘Social science and theevidence-based policy movement’, Social Policy & Society 1:3, pp215-24.64References

Copies of this publication can be obtained from:DfES PublicationsP.O. Box 5050Sherwood ParkAnnesleyNottinghamNG15 0DJTel: 0845 60 222 60Fax: 0845 60 333 60Minicom: 0845 60 555 60Oneline: www.dfespublications.gov.uk© The University of Birmingham 2006Produced by the Department for Education and SkillsISBN 1 84478 776 1Ref No: RR783www.dfes.go.uk/research