A Framework for Evaluating Systems Initiatives

More documents

Recommendations

Info

Identifying Evaluation Options: The Build InitiativeHighest focus Substantial focus Some focus Not yet a focusCONTEXTCOMPONENTSCONNECTIONS INFRASTRUCTURESCALEDesigns/Methods Questions1. Has the initiativechangedthe politicalenvironmentthrough itsactivities?2. Has the initiativeproduced changesto investment,policy, orpractice that willenable changesin components,connections,infrastructure,or scale?• Theory of changeevaluation• Case studies• Public polling• Policy tracking• Key informantsurveys• Coalition analysis• Policymaker/bellwetherinterviews1. Did theinitiativedesign andimplementsystemcomponents asintended?2. Did thecomponentsproduce theirintended impactsfor beneficiaries?1. Did theinitiative designand implementconnectionsand linkages asintended?2. Did theconnections andlinkages producetheir intendedimpacts?(bolded items indicate where the Build evaluation has focused to date)• Program evaluationmethodologies(includingexperimental/quasiexperimental)• Programmonitoring• Quality assessments• Efficiency analyses• Customer surveys• Program evaluationmethodologies(includingexperimental/quasiexperimental)• System mapping• Network analysis• Customer surveys1. Did theinitiativeestablishinfrastructureor supportsthat areconsistent withits objectives?2. Did theinfrastructure orsupports achievetheir objectivesfor effectiveness,sustainability, andquality?• Theory of changeevaluation• Case studies• Performance audits• Managementinformation systems• Practitioner datacollection• While some BuildInitiative stateshave focused onscaling up specificsystemcomponents,scale up at acomprehensivesystems level hasnot yet occurred.• Media tracking29A Framework for Evaluating Systems Initiatives
This example demonstrates how the five-part framework and accompanying menus for bothsystem initiative theories of change and evaluation designs can be used as diagnostic and planningtools. Systems efforts like The Build Initiative are complex. The framework and menus offer waysof managing that complexity so it becomes possible to see more clearly what an initiative is doingand working toward. This then aids decisions about where the evaluation should focus and how itshould look.VII.Concluding Principles for Evaluating Systems InitiativesThis paper grew from the observation that while a great deal of experience already existed on systemsinitiative evaluation, it was difficult to talk clearly about both this field’s past and future becausesystems initiatives are diverse and can mean different things to different people. To address thischallenge, the paper defined the different areas in which initiatives might focus and identified theory ofchange elements and evaluation approaches that fit with each area. The hope is that this frameworkwill provide a clearer way of thinking and talking about systems initiatives, and therefore will facilitatelearning about the field’s past and decision making about its future.While it is clear that no one “right” and generalizable way to approach evaluation has or will emergeto meet all of these challenges, this paper concludes with a set of general principles about whatconstitutes good evaluation practice in this area. 43 Some principles apply to evaluation practicegenerally (but still bear repeating), and others are unique to systems efforts.1) Clarify the evaluation’s audiences and intended uses for the evaluation’s findings.Systems initiatives have many stakeholders, including their funders, implementers, theprograms and institutions that make up the system, and the system’s intended beneficiaries.The evaluation’s primary audiences (or customers) and how they intend to use it should beestablished upfront (e.g., for accountability purposes and determining future funding, forjustifying investments made by showing results and returns on investment, or to identify lessonsthat can be fed back into the initiative for growth and adaptation). If there is more than oneaudience, all audiences’ needs should be considered and balanced. While this point seemsbasic, it is often overlooked, resulting in evaluations that ultimately do not satisfy the audience’sexpectations or needs.Once the audiences are identified, the evaluation generally should involve audience membersin developing the initiative’s theory of change, getting agreement about what outcomes theevaluation will assess within it, and what data will be available and when. Gaining clarity on theevaluation’s purposes upfront can often help avoid misplaced expectations down the road.43Horsch, K. (1997). Evaluating school-linked services: Considerations and best practices. The Evaluation Exchange, 3 (2), 2-4; Shaw, K. (1996).Challenges in evaluating comprehensive school-linked services initiatives. Evaluation Exchange, 1(1), 2-3.30A Framework for Evaluating Systems Initiatives
Page 1 and 2: A Framework For Evaluating Systems
Page 3: Purpose of this PaperThis paper dra
Page 6 and 7: III. Defining Systems Initiatives:
Page 8 and 9: IV. Developing Systems Initiative T
Page 10 and 11: 1 ContextInitiatives focused on con
Page 12 and 13: • New programs or services develo
Page 14 and 15: 4 InfrastructureSometimes a system
Page 16 and 17: OutcomesOutcomes for systems initia
Page 18 and 19: An Evaluation Design Menu for Syste
Page 20 and 21: Because they interact with the poli
Page 22 and 23: ExamplesThe Build Initiative in Pen
Page 24 and 25: 4 Evaluating InfrastructureSystems
Page 26 and 27: Evaluation MethodologiesEvaluating
Page 28 and 29: VI. Applying the Framework: The Bui
Page 32 and 33: 2) Base evaluation decisions on the
Page 34 and 35: 8) Hold some systems initiatives, b
Page 36: www.buildinitiative.orgThe Heinz En

A Framework for Evaluating Systems Initiatives

Create successful ePaper yourself

Delete template?

Save as template?