A Framework for Evaluating Systems Initiatives

ExamplesThe North Carolina Smart Start Technical Assistance Center evaluation provides an example for this focusarea. The Center provides infrastructure development assistance to many states trying to build theirown early childhood development systems. Being completed in two phases, the evaluation is exploringversions of both evaluation questions described above.The evaluation’s first phase studied the effectiveness of the Center’s technical assistance in helpingstates to meet their system building goals. In other words, it looked at whether states were successfulin meeting their infrastructure goals and the role they thought technical assistance played in theirachievements. The evaluation’s second phase will include a qualitative study of how technical assistanceaccelerated states’ efforts to improve positive impacts for children. This work builds on North Carolina’sown success in establishing, expanding, and sustaining Smart Start as an early childhood systemsinitiative in the state.5Evaluating ScaleSystems initiatives focused on scale attempt to ensure that a comprehensive and quality system isavailable to an increasing (and eventually complete) number of intended beneficiaries.Evaluation QuestionsEvaluations in this area generally concentrate on two key questions:1) Did the initiative enable system scale up with quality and fidelity?2) Did scale up result in broad impacts for beneficiaries at a system-wide population level?First, evaluations can explore the scale-up process itself, asking whether the system is being scaledup with fidelity and quality, for example. Because there are many historical examples of systemsinitiatives that “demonstrated, researched, and died” during scale up, or that experienced modeldrift and ended up offering lesser services and benefits than intended, evaluations that focus on thisprocess and continuously feed back findings to inform midcourse corrections can be critical to theinitiative’s success. 39Second, evaluations can assess the system and its impacts for beneficiaries. The main evaluationquestion here is whether systems are producing comprehensive impacts for the people they serve.Evaluations of initiatives with a scale-up focus share a lot in common with the traditional programevaluation approaches described in the earlier section about evaluating initiatives focusedon components. The difference is that here the system, rather than a single program, is the“intervention.” Viewed in this way, evaluations that address the first question about the scale-upprocess are equivalent to program monitoring or process evaluations; those that address the secondquestion are equivalent to program impact assessments.39Weiss, H. (1995). New approaches to evaluating systems: The Gwen R. Iding Brogden Distinguished Lecture Series (p.412). In Liberton, C., Kutash,K., & Friedman, R. (Eds.), 8th Annual Research Conference Proceedings on A System of Care for Children’s Mental Health: Expanding theResearch Base. Tampa, FL: The Research and Training Center for Children’s Mental Health.24A Framework for Evaluating Systems Initiatives