A Framework for Evaluating Systems Initiatives

More documents

Recommendations

Info

An Evaluation Design Menu for Systems InitiativesCONTEXTCOMPONENTSCONNECTIONS INFRASTRUCTURESCALE1. Has the initiative1. Did the initiative1. Did the initiative1. Did the initiative1. Did the initiativechanged thedesign anddesign andestablishenable system scalepoliticalimplement systemimplementinfrastructure orup with quality andenvironmentcomponents asconnectionssupports that arefidelity?Methodologies Questionsthrough itsactivities?2. Has the initiativeproduced changesto investment,policy, or practicethat will enablechanges incomponents,connections,infrastructure,or scale?• Theory of changeevolution• Case studies• Public polling• Policy tracking• Key informantsurveys• Coalition analysis• Policy/bellwetherinterviews• Media trackingconsistent with itsobjectives?2. Did theInfrastructure orsupports achievetheir objectivesfor effectiveness,sustainability, andquality?• Theory of changeevaluation• Case studies• Performance audits• Managementinformation systems• Practitioner datacollectionintended?2. Did the componentsproduce theirintended impactsfor beneficiaries?• Program evaluationmethodologies(includingexperimental/quasiexperimental)• Program monitoring• Quality assessments• Efficiency analyses• Customer surveysand linkages asintended?2. Did the connectionsand linkagesproduce theirintended impacts?• Program evaluationmethodologies(includingexperimental/quasiexperimental)• System mapping• Network analysis• Customer surveys2. Did scale up resultIn broad impactsfor beneficiariesat a system-widepopulation level?• Population-baseddemographic andservice analysis• Program evaluationmethodologies(includingexperimental/quasiexperimental)• System/programmonitoring• Results-basedaccountability17A Framework for Evaluating Systems Initiatives
Evaluation questions, designs, and methods can be “mixed and matched” as appropriate. For example,a quasi-experimental design may co-exist with a theory of change approach, or case studies may beused alongside a results-based accountability approach. Initiative evaluations can incorporate multipledesigns and a wide range of data collection methods.The remainder of this section describes evaluation options for each focus area in more detail, discussingrelevant evaluation questions and methodologies. In addition, it offers information about theevaluations of the real-life initiatives described earlier.1Evaluating ContextSystems initiatives focused on context attempt to affect the political environment so it better supportssystems’ development and success.Evaluation QuestionsThe key evaluation questions for these initiatives are:1) Has the initiative changed the political environment through its activities?2) Has the initiative produced changes to investment, policy, or practice that will enable changes incomponents, connections, infrastructure, or scale?Evaluation MethodologiesInitiatives with a context focus generally use a theory of change approach to evaluation. Once atheory is developed for how outcomes in the political context will be achieved, evaluators seekempirical evidence that the theory’s components are in place and that the theorized links betweenthem exist. In other words, they compare the theory with actual experience. 27 Evaluators “constructmethods for data collection and analysis to…examine the extent to which program [or initiative]theories hold. The evaluation should show which of the assumptions underlying the program breakdown, where they break down, and which of the several theories underlying the program are bestsupported by the evidence.” 28Mostly, evaluations that use this approach will not attempt to determine whether the theory’scomponents are causally linked. Rather, they assess the components separately, using multiple andtriangulated methods where possible, and then use both data and evaluative judgment to determinewhether a plausible and defensible case be made that the theory of change worked as anticipated andthat the initiative had its intended effects. As the Urban Health Initiative evaluation example belowdemonstrates, however, it is possible to incorporate design elements like comparison groups or othercounterfactuals into a theory of change approach to strengthen confidence in interpretations about thelinks between theory of change components.27Connell, J., Kubisch, A., Schorr, L., & Weiss, C. (1995) (Eds.). New approaches to evaluating community initiatives: Concepts, methods, and contexts.Washington, DC: The Aspen Institute.28Weiss, C. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children andfamilies (p.67). In Connell, J., Kubisch, A., Schorr, L., & Weiss, C. (Eds.) New approaches to evaluating community initiatives: Concepts, methods,and contexts. Washington, DC: The Aspen Institute.18A Framework for Evaluating Systems Initiatives
Page 1 and 2: A Framework For Evaluating Systems
Page 3: Purpose of this PaperThis paper dra
Page 6 and 7: III. Defining Systems Initiatives:
Page 8 and 9: IV. Developing Systems Initiative T
Page 10 and 11: 1 ContextInitiatives focused on con
Page 12 and 13: • New programs or services develo
Page 14 and 15: 4 InfrastructureSometimes a system
Page 16 and 17: OutcomesOutcomes for systems initia
Page 20 and 21: Because they interact with the poli
Page 22 and 23: ExamplesThe Build Initiative in Pen
Page 24 and 25: 4 Evaluating InfrastructureSystems
Page 26 and 27: Evaluation MethodologiesEvaluating
Page 28 and 29: VI. Applying the Framework: The Bui
Page 30 and 31: Identifying Evaluation Options: The
Page 32 and 33: 2) Base evaluation decisions on the
Page 34 and 35: 8) Hold some systems initiatives, b
Page 36: www.buildinitiative.orgThe Heinz En

A Framework for Evaluating Systems Initiatives

Create successful ePaper yourself

Delete template?

Save as template?