A Framework for Evaluating Systems Initiatives

Evaluation questions, designs, and methods can be “mixed and matched” as appropriate. For example,a quasi-experimental design may co-exist with a theory of change approach, or case studies may beused alongside a results-based accountability approach. Initiative evaluations can incorporate multipledesigns and a wide range of data collection methods.The remainder of this section describes evaluation options for each focus area in more detail, discussingrelevant evaluation questions and methodologies. In addition, it offers information about theevaluations of the real-life initiatives described earlier.1Evaluating ContextSystems initiatives focused on context attempt to affect the political environment so it better supportssystems’ development and success.Evaluation QuestionsThe key evaluation questions for these initiatives are:1) Has the initiative changed the political environment through its activities?2) Has the initiative produced changes to investment, policy, or practice that will enable changes incomponents, connections, infrastructure, or scale?Evaluation MethodologiesInitiatives with a context focus generally use a theory of change approach to evaluation. Once atheory is developed for how outcomes in the political context will be achieved, evaluators seekempirical evidence that the theory’s components are in place and that the theorized links betweenthem exist. In other words, they compare the theory with actual experience. 27 Evaluators “constructmethods for data collection and analysis to…examine the extent to which program [or initiative]theories hold. The evaluation should show which of the assumptions underlying the program breakdown, where they break down, and which of the several theories underlying the program are bestsupported by the evidence.” 28Mostly, evaluations that use this approach will not attempt to determine whether the theory’scomponents are causally linked. Rather, they assess the components separately, using multiple andtriangulated methods where possible, and then use both data and evaluative judgment to determinewhether a plausible and defensible case be made that the theory of change worked as anticipated andthat the initiative had its intended effects. As the Urban Health Initiative evaluation example belowdemonstrates, however, it is possible to incorporate design elements like comparison groups or othercounterfactuals into a theory of change approach to strengthen confidence in interpretations about thelinks between theory of change components.27Connell, J., Kubisch, A., Schorr, L., & Weiss, C. (1995) (Eds.). New approaches to evaluating community initiatives: Concepts, methods, and contexts.Washington, DC: The Aspen Institute.28Weiss, C. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children andfamilies (p.67). In Connell, J., Kubisch, A., Schorr, L., & Weiss, C. (Eds.) New approaches to evaluating community initiatives: Concepts, methods,and contexts. Washington, DC: The Aspen Institute.18A Framework for Evaluating Systems Initiatives