A Framework for Evaluating Systems Initiatives

More documents

Recommendations

Info

• New programs or services developed within the system• Expanded individual program reach or coverage• Improved program quality• Increased program operational efficiency• Beneficiary outcomes that precede impactImpactsSystems initiatives focused on individual programs may be expected to demonstrate measurableimpacts for program participants. For example, a nutrition program would expect better health resultsfor participants. In contrast to evaluations of more comprehensive systems where impacts are expectedacross a broad array of domains, expected impacts here are specific to the program or interventionbeing evaluated.ExampleIn addition to its focus on context, The Build Initiative focuses on system components. Whileattempting to make improvements in the political environment that enable early childhooddevelopment systems to be built, Build states also work to improve the individual subsystems in them.This step is essential to ensuring that children and families have access to the comprehensive set ofprograms and services they need to be successful. For example, some states are working on enhancingtheir early and education subsystem by making high-quality preschool programs available for all threeandfour-year-olds. Others are concentrating on creating infant mental health programs and services.Still others are developing strategies to strengthen the quality of programs that currently exist. Whileall of the states have a broader system building vision and an accompanying strategy focused primarilyon political context, they take advantage of opportunities to advance the development of specificsubsystems and see this work as very much a part of their overall systems efforts.3ConnectionsSystems initiatives that concentrate on connections focus on what makes a system a system—theintegration, linkages, and alignment between its parts.ActivitiesInitiatives focused on this area attempt to connect subsystems or programs in meaningful ways. Onthe frontlines, those connections may include, for example, programs across or within subsystemsthat share professional development, staffing, facilities, technology and communication, datacollection, or funding. At the administrative level, such connections may include, for example, alignedeligibility requirements and enrollment processes, streamlined reporting procedures, coordinatedcase management, and established protocols and memoranda of understanding for referrals acrosssubsystems. These kinds of connections ensure that, when necessary, beneficiary needs identified inone subsystem can be referred to and managed by another.11A Framework for Evaluating Systems Initiatives
OutcomesOutcomes here relate to connections themselves in the form of increased system coordination,alignment, integration, or linkages. The challenge is operationalizing these concepts so they aremeasurable and relevant to the systems initiative at hand. Ways of operationalizing connectionsinclude, for example:• Coordinated eligibility assessments• Referrals occurring from one program to another• Activities to ease within-system transitions• Joint planning across system components• Cross-system competencies or skills standards• Cross-system training• Shared data systems for tracking individuals• Memoranda of agreements between system componentsImpactsSystems initiatives focused on connections may be expected to demonstrate impacts for systembeneficiaries. Better results for beneficiaries are expected when the system is connected comparedto when it is not. The specific individual-level impacts measured will depend on the level at whichconnections are expected. If many connections are expected across the system, then impacts can beexpected across a wide array of domains (e.g., health, literacy, mental health, etc.). If they are expectedwithin or between particular subsystems, then expected impacts should be more specific to those areas.ExampleThe W.K. Kellogg Foundation’s SPARK Initiative focuseson connections. It funds partnerships in seven statesand the District of Columbia to ensure that children areready for school and that schools are ready for children.SPARK (Supporting Partnerships to Assure Ready Kids)sites have adopted alignment models that take intoaccount schools’ relationships with the community andwith the early care and education system. They supportseamless transitions from early childhood through elemenschool by aligning these systems with health and other sersystems. SPARK is demonstrating that it is possible for theand education system to connect with nontraditional parthealth on behalf of kids. 1717W.K. Kellogg Foundation. SPARK overview. Retrieved on June 29, 2007 fromhttp://www.wkkf.org/default.aspx?tabid=75&CID=168&NID=61&LanguageID=0.12A Framework for Evaluating Systems Initiatives
Page 1 and 2: A Framework For Evaluating Systems
Page 3: Purpose of this PaperThis paper dra
Page 6 and 7: III. Defining Systems Initiatives:
Page 8 and 9: IV. Developing Systems Initiative T
Page 10 and 11: 1 ContextInitiatives focused on con
Page 14 and 15: 4 InfrastructureSometimes a system
Page 16 and 17: OutcomesOutcomes for systems initia
Page 18 and 19: An Evaluation Design Menu for Syste
Page 20 and 21: Because they interact with the poli
Page 22 and 23: ExamplesThe Build Initiative in Pen
Page 24 and 25: 4 Evaluating InfrastructureSystems
Page 26 and 27: Evaluation MethodologiesEvaluating
Page 28 and 29: VI. Applying the Framework: The Bui
Page 30 and 31: Identifying Evaluation Options: The
Page 32 and 33: 2) Base evaluation decisions on the
Page 34 and 35: 8) Hold some systems initiatives, b
Page 36: www.buildinitiative.orgThe Heinz En

A Framework for Evaluating Systems Initiatives

Create successful ePaper yourself

Delete template?

Save as template?