A Framework for Evaluating Systems Initiatives

More documents

Recommendations

Info

VI. Applying the Framework: The Build Initiative The last two sections demonstrate how the five focus areas can act as a framework for definingsystems initiatives, constructing their theories of change, and designing their evaluations. The five-partframework offers a way to break down the initiative’s complexities into more manageable parts withoutlosing sight of “the whole.”This section demonstrates how the framework can be applied to an actual systems initiative, with TheBuild Initiative serving as the example. The framework is applied to The Build Initiative in three ways;to: 1) map Build’s focus areas (at an overall initiative level, across Build states), 2) define Build’s relevanttheory of change elements, and 3) identify Build’s existing and potential evaluation options.Mapping the areas of focusThe first step is to map The Build Initiative’s focus areas and their relative emphases. This exercisesets expectations about what the initiative is trying to achieve and therefore signals where theevaluation should place more or less focus. The shaded figure below shows where the Initiativecurrently is focused.CONTEXT COMPONENTS CONNECTIONS INFASTRUCTURE SCALEHighest focus Substantial focus Some focus Not yet a focusTo date, The Build Initiative has focused on four of the five areas—context, components, connections,and infrastructure. Of these four areas, Build states generally place the most emphasis on changingthe political context to better support systems. While the Initiative’s goal is to work toward scale up,no Build states are yet at the point where they are scaling up comprehensive early childhood systems.Defining the theory of change elementsThe second step is to define the theory of change elements associated with The Build Initiative’s focusareas, so that the theory can further guide evaluation planning. The figure on page 28 identifiesBuild theory of change elements. It does not list all possible activities and outcomes; it pulls itemsfrom the theory of change menu offered earlier and identifies examples of the kinds of activities andoutcomes on which many Build states focus.Identifying evaluation optionsThe third step is to identify relevant evaluation options given the Initiative’s theory of change. Thefigure on page 28 uses the evaluation design menu as a guide and bolds the elements that thenational Build evaluation and local evaluation partners have focused on to date. The figure illustratesboth where the evaluation is focused now and options for where it could focus in the future.Note that this step necessarily requires strategic choices about what the evaluation will and willnot do. For example, rarely are enough resources available to collect data on every outcome in thetheory of change. Methodological choices should be based on what information the evaluation’saudiences want, where the initiative currently is in its development, the evaluation’s timeframe, andthe evaluation resources available.27A Framework for Evaluating Systems Initiatives
Defi ning Theory of Change Elements: The Build InitiativeHighest focus Substantial focus Some focus Not yet a focusCONTEXTCOMPONENTSCONNECTIONS INFRASTRUCTURESCALEImpacts Outcomes (examples) Activities (examples)• Developing a vision• Educatingpolicymakers• Engaging the public• Engaging the media• Doing electronicoutreach• Evidence base aboutthe system and itsneeds• Shared earlylearning vision• Gubernatorialand other politicalleadership• New and mobilizedearly childhoodadvocates• Changed policies(e.g., increasedfunding forpreschool; increasedchild care subsidyrates)• Expanding highqualityprograms• Developing newprograms orservices• Expandingeligibility• Increasing accessand availability• Expanded programs(e.g., universalpreschool, earlyintervention)• New programs(e.g., infant mentalhealth services)• Better access (e.g.,expanded SCHIP orMedicaid eligibility)• Higher-qualityprograms(e.g., child care)• Creating forumsfor cross-sectorplanning• Smoothingdevelopmentaltransitions• Connecting datasystems• Promotingcollaboration andreferrals• Improvedtransitions (e.g.,2- and 4-yearcollege articulation)• Joint training(e.g., for preschooland Kindergartenteachers)• Co-locatedprograms (e.g.,“One-stop” familyresource centers)• Increased referrals(e.g., pediatriciansto earlyintervention)• Developingnew governanceproposals• Improving quality(quality ratingsystems, standards)• Improving statelocalconnections• Developingbetter two-waycommunications• Monitoringthe system• New earlychildhoodgovernancestructures• Quality ratingsystems (e.g., forchild care)• Assessments(e.g., Kindergartenreadiness)• Standards/guidelines (e.g.,infant-toddler)• Local infrastructure(e.g., localgovernancestructures)• Outcomes are not • Improved child and • Improved child • Outcomes are notexpected to show family outcomes in and family expected to showdirect impacts for relevant domains— outcomes where direct impacts forchildren and health, literacy, systemchildren andfamilies mental health, etc.componenets areconnectedfamilies• While some BuildInitiative stateshave focused onscaling up specificsystemcomponents,scale up at acomprehensivesystems level hasnot yet occurred.28A Framework for Evaluating Systems Initiatives
Page 1 and 2: A Framework For Evaluating Systems
Page 3: Purpose of this PaperThis paper dra
Page 6 and 7: III. Defining Systems Initiatives:
Page 8 and 9: IV. Developing Systems Initiative T
Page 10 and 11: 1 ContextInitiatives focused on con
Page 12 and 13: • New programs or services develo
Page 14 and 15: 4 InfrastructureSometimes a system
Page 16 and 17: OutcomesOutcomes for systems initia
Page 18 and 19: An Evaluation Design Menu for Syste
Page 20 and 21: Because they interact with the poli
Page 22 and 23: ExamplesThe Build Initiative in Pen
Page 24 and 25: 4 Evaluating InfrastructureSystems
Page 26 and 27: Evaluation MethodologiesEvaluating
Page 30 and 31: Identifying Evaluation Options: The
Page 32 and 33: 2) Base evaluation decisions on the
Page 34 and 35: 8) Hold some systems initiatives, b
Page 36: www.buildinitiative.orgThe Heinz En

A Framework for Evaluating Systems Initiatives

Create successful ePaper yourself

Delete template?

Save as template?