Local Evaluation of Children's Services Learning from the Children's ...

legitimacy of qualitative methods and multiple (quantitative and qualitative) evaluationmethods is being increasingly recognised among UK policy makers as an appropriateapproach to evaluating complex interventions (Coote, et al., 2004; Spencer, et al.,2004). Indeed, as outlined in Chapter 2 of this report, local evaluators have tended tofully embrace multiple methods, and hence recognise that different methods aremore appropriate in examining different elements of a programme’s work. Anevaluator from a Wave One partnership, for example, described how the partnershipboard tended to favour ‘hard’ quantitative evidence rather than qualitative evaluationdata, and found it difficult to accept the legitimacy and relevance of the latter:It might just be some kind of headline statistics they want really…I know thatthe drop-in [project] is a life-line to some of those parents that attend there, theywouldn’t know what to do with their children if they didn’t go to that drop-in, andI can write that up but it’s not hard evidence, just the stats really whichsometimes doesn’t really tell the whole story… And even though we might havesay all the stats they’re wanting, what the actual people on the ground mightsay might be something completely different.Similarly, a Wave Two evaluation described how partnership board members’ limitedunderstandings of notions such as indicators and impacts had been a frustratingbarrier to progress in terms of their unrealistic expectations of the evaluation:The partnership board didn’t really understand... why can’t you show how gooda service is?... And I said, well actually, if you haven’t got those indicators inplace, I can’t magic them out of the air. And you need to have them on anongoing basis, because I can’t do them in retrospect, because you should havebeen recording what risk factor the children came in with, effectively what’sbeen done with them and what they came out with. I can do what they cameout with, but I can’t compare that to anything at that point of entry.Many of the reports also remained tentative in offering definitive findings on progress,and have given observations that can continue to inform an evolving programme thatis formative. This has been because local evaluators have had to take account oftime lapses in the expected progress of programmes resulting from problems ofgetting programmes started in some parts of the country, and funding uncertaintiesencountered by programmes that have slowed progress. Additionally, localevaluators have produced final reports when some programmes are about half waythrough their lives resulting in evaluations being short or medium term in duration. Asa consequence, local evaluators have been reluctant to identify lasting outcomes.Hence, there can be a tension between providing evaluation outputs that areappropriate for readers in terms of being short and using accessible terms asopposed to fully evidencing statements made and fully exploring complex issues.44Chapter 3