A Framework for Evaluating Systems Initiatives

More documents

Recommendations

Info

OutcomesOutcomes for systems initiatives focused on scale depend on how scale is defined. Scale definitionsmay include:• System spread• System depth• System sustainability• Shifts in system ownershipFor health, education, or human service systems, traditional definitions of scale concentrate on thesystem’s spread, or increases in the number of programs and people served. But this definition alone islimited; it says little about how scale up has affected the services people receive. The concept of spreadshould expand beyond simply “upping the numbers” to include the spread of ideas, beliefs, values,and principles that support bringing systems to scale. Depth, or the nature and quality of change thattakes place at the practice level, is another important dimension of scale, as is sustainability or whetherthe system can be maintained over time. Finally, the definition of scale can include a shift in ownership,meaning that a broad array of stakeholders, especially those on the frontlines, assume responsibility formaintaining the scaled-up system. 23Outcomes also may include various benchmarks for longitudinal impact measures (e.g., expectingincremental improvements in low birth weight statistics over time), or the individual-level outcomesthat precede impacts (e.g., more mothers receiving prenatal care contributing to better birth weightstatistics).ImpactsSystems initiatives that concentrate on scale generally are expected to demonstrate impacts for systembeneficiaries. Because the system is, by definition, comprehensive, these impacts should exist across awide range of domains.ExamplesThe First 5 California School Readiness Initiative is an example of a systems initiative focused on scale.Supported by funding from a tobacco tax passed in 1998, it is the largest early childhood and schoolreadiness initiative in the nation (over $450 million has been allocated since 2002). The initiativeprovides funding to all of California’s counties for the development and functioning of communitybasedearly childhood systems that provide and coordinate high-quality programs for children agesbirth to five in areas that include early care and education, parenting and family support, health andsocial services, and school capacity. This initiative falls into the scale category because it providesfunding for movement toward comprehensive early childhood development systems statewide. Thesesystems are expected to be comprehensive and accessible enough to produce positive changes in abroad spectrum of child and family outcomes that relate to school readiness. 2423Coburn, C. (2003). Rethinking scale: Moving beyond the numbers to deep and lasting change. Educational Researcher, 32(6), 3-12.24First 5 California. First 5 California School Readiness Initiative. Retrieved on June 30, 2007 from http://www.ccfc.ca.gov/SchoolReady1.htm.15A Framework for Evaluating Systems Initiatives
North Carolina’s Smart Start is another example in the scale category. This now 14-year-old initiativewas established to build a comprehensive and coordinated early childhood system statewide forpurposes of improving school readiness outcomes. The initiative offers children birth to age five arange of services with the goal of enabling children to enter school healthy and ready for success.Smart Start is funded through public dollars drawn from the state’s general fund that are then directedto local partnerships in all of North Carolina’s 100 counties (the state appropriation was $204 millionin fiscal year 2007; funding peaked at $231 million in fiscal year 2000). 25 While the majority of SmartStart funding is spent on child care or child care-related services (e.g., subsidies), about one-third oflocal partnership dollars must address other issues affecting young children and their families, such asfamily support and health care. 26V. Designing Systems Initiative EvaluationsAs the last section makes clear, all systems initiatives are not the same. It follows then that theirevaluations should not be the same. Systems initiative evaluations should be tailored to their uniquetheories of change, assessing the outcomes and impacts connected to the parts of the system they areattempting to change.The figure on the next page was developed to guide evaluation planning decisions for systemsinitiatives. Like the theory of change menu, it is organized around the five focus areas and offers ideasabout appropriate evaluation choices for initiatives that incorporate each.Each focus area features two evaluation questions that generally address:• Did the initiative do what it said it would do (in that focus area)?• Did the initiative produce the expected results (for that focus area)?The fi gur e also features possible evaluation methodologies to address those questions. These lists ofhods are not exhaustive or prescriptive. They offer ideas, but do not spell outtions should look. Those decisions must be based on a careful considerationecific initiative in question and what it is designed to do., it is important to keep in mind that most systems initiatives are hybrids ands on more than one area. Consequently, multiple evaluation questions willy and evaluations should feature a mix of design elements to address thoseions. Additionally, evaluations should identify the areas in which the initiatives the most emphasis, and plan for similar levels of evaluation emphasis inse same areas.for Law and Social Policy. North Carolina Smart Start. Retrieved on June 30, 2007 fromww.clasp.org/ChildCareAndEarlyEducation/map030707nc1.htm.n, J., Stover Wright, M., & Bruner, C. (2006). Beyond parallel play: Emerging state and community planningbuilding early learning. Des Moines, IA: State Early Childhood Policy Technical Assistance Network16A Framework for Evaluating Systems Initiatives
Page 1 and 2: A Framework For Evaluating Systems
Page 3: Purpose of this PaperThis paper dra
Page 6 and 7: III. Defining Systems Initiatives:
Page 8 and 9: IV. Developing Systems Initiative T
Page 10 and 11: 1 ContextInitiatives focused on con
Page 12 and 13: • New programs or services develo
Page 14 and 15: 4 InfrastructureSometimes a system
Page 18 and 19: An Evaluation Design Menu for Syste
Page 20 and 21: Because they interact with the poli
Page 22 and 23: ExamplesThe Build Initiative in Pen
Page 24 and 25: 4 Evaluating InfrastructureSystems
Page 26 and 27: Evaluation MethodologiesEvaluating
Page 28 and 29: VI. Applying the Framework: The Bui
Page 30 and 31: Identifying Evaluation Options: The
Page 32 and 33: 2) Base evaluation decisions on the
Page 34 and 35: 8) Hold some systems initiatives, b
Page 36: www.buildinitiative.orgThe Heinz En

A Framework for Evaluating Systems Initiatives

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?