GAO-12-208G, Designing Evaluations: 2012 Revision

Designs for Different
Types of Programs
Chapter 4: Designs for Assessing Program
Implementation and Effectiveness
• Evaluations of program effectiveness also measure key program
components to help learn why a program is not working as well as
was expected.
An evaluation may find that a program failed to achieve its intended
outcomes for a variety of reasons, including: incomplete or poor quality
implementation of the program; problems in obtaining valid and reliable
data from the evaluation; environmental influences that blunt the
program’s effect; or the ineffectiveness of the program or intervention for
the population and setting in which it was tested. Thus, examination of
program implementation is very important to interpreting the results on
outcomes. Moreover, because an impact evaluation may be conducted in
a restricted range of settings in order to control for other influences on
outcomes, its findings may not apply to other settings or subgroups of
recipients. Thus, it is important to test the program or intervention’s
effects in several settings or under various circumstances before drawing
firm conclusions about its effectiveness. A formal synthesis of the findings
of multiple evaluations can provide important information about the
limitations on—or factors influencing—program impacts, and be
especially helpful in learning what works for whom and under what
circumstances.
As evaluation designs are tailored to the nature of the program and the
questions asked, it becomes apparent that certain designs are
necessarily excluded for certain types of programs. This is particularly
true of impact evaluations because of the stringent conditions placed on
the evidence needed to draw causal conclusions with confidence.
Experimental research designs are best adapted to assess discrete
interventions under carefully controlled conditions in the experimental
physical and social sciences. The federal government has only relatively
recently expanded its efforts to assess the effectiveness of all federal
programs and policies, many of which fail to meet the requirements for
successful use of experimental research designs.
To assist OMB officials in their efforts to assess agency evaluation efforts,
an informal network of federal agency evaluators provided guidance on
the relevance of various evaluation designs for different types of federal
programs. Table 5 summarizes the features of the designs discussed in
this chapter as well as the types of programs employing them.
Page 46 GAO-12-208G