A Framework for Evaluating Systems Initiatives

5) Factor investment levels for both systems initiatives and their evaluations intoevaluation decisions. Just as systems initiatives differ in focus, they also differ in funding andtimeframe. Initiative funding can range from the hundreds of thousands to hundreds of millionsand from a timeframe of one or two years to ten years or more. Expectations about what resultsinitiatives will produce should be adjusted accordingly.Similarly, initiative funding directly affects the resources available for evaluation. Standards aboutrigor or the “level of evidence” evaluations are expected to produce should be considered in lightof their funding, as the amount of evaluation resources available greatly affects methodologicalchoices. Generally, it is wise to conduct an evaluability assessment that assesses the proposedevaluation’s likely benefits against the cost and level of effort it creates.6) Establish the necessary timeframe for results. Many systems initiative impacts will takeNumerous years to accomplish. This is particularly true for initiatives where the ultimate goalis better results for individuals such as children. These impacts may take years to show up (ifthey are even appropriate to measure in the first place). For example, school readiness impactsconnected to early childhood development systems may require several years to show up and mayonly be established with real confidence when children reach third grade or higher. Evaluationplans should be realistic and expectations about results aligned with the initiative’s theory ofchange, its timeframe, and its scope and depth. What data to expect and when (at least roughly)should be clear to the evaluation’s audience. The fact that both those conducting systemsinitiatives and their evaluators will likely be pressured to “show impact” at some point beforesuch results may be ready, underscores the importance of adhering to this principle both early onand throughout the evaluation.In particular, systems initiatives should not be accountable for producing beneficiary impacts simplybecause participants agree that they would like to see such impacts occur. Measurable changesin such impacts should be expected only when a systems initiative makes investments in strategiesugh to produce them (if the theory of change’s assumptions hold).ure and value interim outcomes. In the context of health, education, orservices, all systems initiatives have their eyes on the ultimate prize—betters for the system’s intended beneficiaries. However, as stated above, thesean take many years to achieve and may involve major new investments orin the current system’s operations. It is important to identify outcomesthe stage for longer-term impact, and then to avoid assigning a lesser-classthose outcomes. While better impacts for beneficiaries should remainte goal and initiatives should be clear about how they contribute to thater outcomes related to the systems initiative strategy and linked to context,ents, connections, infrastructure, or scale are important measures ofs along the way.32A Framework for Evaluating Systems Initiatives