A Framework for Evaluating Systems Initiatives

4 Evaluating InfrastructureSystems initiatives focused on infrastructure attempt to ensure the system has the supports it needs tofunction effectively and with quality. Better system supports may be needed, for example, in the areasof governance, financing, data systems, and practitioner support or workforce development.Evaluation QuestionsEvaluations of infrastructure-focused initiatives explore two main questions:1) Did the initiative establish infrastructure or supports that are consistent with its objectives?2) Did the infrastructure or supports achieve its objectives for effectiveness, sustainability, or quality?Evaluation MethodologiesEvaluations that address the first question of whether the system has the supports it needs may use atheory of change approach. The theory of change would articulate the strategy for how a particularinfrastructure change is expected to occur and then use different methods to test whether the theoryof change is working as intended. Theories of change for initiatives focused on infrastructure mayshare characteristics in common with initiatives focused on context, as infrastructure changes oftenrequire policy changes.Case studies offer another methodological option for these initiatives. Case studies use multiplemethods (quantitative or qualitative) to examine how an infrastructure initiative played out and thefactors that contributed to its success or lack thereof. For example, the Success Case Method is aparticular type of case study that is relatively cost-effective. It is a post-hoc analysis of success (orfailure) that combines systematic case study methodology with storytelling, and reports results thatstakeholders can easily understand and believe. 38Evaluations focused on the second question of whether supports are effective once they are in placeexamine supports’ benefits for the system. For example, an evaluation might examine whether lesscategorical and more flexible funding actually results in more comprehensive and connected services atthe local level. Or an evaluation might examine how practitioner supports in the form of education ortraining benefit the system’s workforce.These evaluations also can draw on a range of methodological options. For example, a performanceaudit can determine how well a governance entity or state agency is functioning. Customersatisfaction surveys can determine whether particular supports are accessible and user-friendly.Program evaluations can determine whether new education and training programs or policies have animpact on the workforce.38Brinkerhoff, R. (2003). The success case method: Find out quickly what’s working and what’s not. San Francisco, CA: Berrett-Koehler.23A Framework for Evaluating Systems Initiatives