A Framework for Evaluating Systems Initiatives

This example demonstrates how the five-part framework and accompanying menus for bothsystem initiative theories of change and evaluation designs can be used as diagnostic and planningtools. Systems efforts like The Build Initiative are complex. The framework and menus offer waysof managing that complexity so it becomes possible to see more clearly what an initiative is doingand working toward. This then aids decisions about where the evaluation should focus and how itshould look.VII.Concluding Principles for Evaluating Systems InitiativesThis paper grew from the observation that while a great deal of experience already existed on systemsinitiative evaluation, it was difficult to talk clearly about both this field’s past and future becausesystems initiatives are diverse and can mean different things to different people. To address thischallenge, the paper defined the different areas in which initiatives might focus and identified theory ofchange elements and evaluation approaches that fit with each area. The hope is that this frameworkwill provide a clearer way of thinking and talking about systems initiatives, and therefore will facilitatelearning about the field’s past and decision making about its future.While it is clear that no one “right” and generalizable way to approach evaluation has or will emergeto meet all of these challenges, this paper concludes with a set of general principles about whatconstitutes good evaluation practice in this area. 43 Some principles apply to evaluation practicegenerally (but still bear repeating), and others are unique to systems efforts.1) Clarify the evaluation’s audiences and intended uses for the evaluation’s findings.Systems initiatives have many stakeholders, including their funders, implementers, theprograms and institutions that make up the system, and the system’s intended beneficiaries.The evaluation’s primary audiences (or customers) and how they intend to use it should beestablished upfront (e.g., for accountability purposes and determining future funding, forjustifying investments made by showing results and returns on investment, or to identify lessonsthat can be fed back into the initiative for growth and adaptation). If there is more than oneaudience, all audiences’ needs should be considered and balanced. While this point seemsbasic, it is often overlooked, resulting in evaluations that ultimately do not satisfy the audience’sexpectations or needs.Once the audiences are identified, the evaluation generally should involve audience membersin developing the initiative’s theory of change, getting agreement about what outcomes theevaluation will assess within it, and what data will be available and when. Gaining clarity on theevaluation’s purposes upfront can often help avoid misplaced expectations down the road.43Horsch, K. (1997). Evaluating school-linked services: Considerations and best practices. The Evaluation Exchange, 3 (2), 2-4; Shaw, K. (1996).Challenges in evaluating comprehensive school-linked services initiatives. Evaluation Exchange, 1(1), 2-3.30A Framework for Evaluating Systems Initiatives