GAO-12-208G, Designing Evaluations: 2012 Revision

Table 3: Common Designs for Outcome Evaluations
Chapter 4: Designs for Assessing Program
Implementation and Effectiveness
Evaluation question Design
Is the program achieving its desired outcomes or having other
important side effects?
Do program outcomes differ across program components,
providers or recipients?
Assessing the Achievement
of Intended Outcomes
Assessing Change in
Outcomes
Source GAO.
• Compare program performance to law and regulations,
program logic model, professional standards, or
stakeholder expectations
• Assess change in outcomes for participants before and
after exposure to the program
• Assess differences in outcomes between program
participants and nonparticipants
Assess variation in outcomes (or change in outcomes) across
approaches, settings, providers, or subgroups of recipients
Like outcome monitoring, outcome evaluations often assess the benefits
of the program for participants or the broader public by comparing data on
program outcomes to a preestablished target value. The criterion could
be derived from law, regulation, or program design, while the target value
might be drawn from professional standards, stakeholder expectations, or
the levels observed previously in this or similar programs. This can help
ensure that target levels for accomplishments, compliance, or absence of
error are realistic. For example,
• To assess the immediate outcomes of instructional programs, an
evaluator could measure whether participants’ experienced short-term
changes in knowledge, attitudes, or skills at the end of their training
session. The evaluator might employ post-workshop surveys or
conduct observations during the workshops to document how well
participants understood and can use what was taught. Depending on
the topic, industry standards might provide a criterion of 80 percent or
90 percent accuracy, or demonstration of a set of critical skills, to
define program success. Although observational data may be
considered more accurate indicators of knowledge and skill gains than
self-report surveys, they can often be more resource-intensive to
collect and analyze.
In programs where there are quantitative measures of performance but
no established standard or target value, outcome evaluations at least may
rely on assessing change or differences in desired outputs and outcomes.
The level of the outcome of interest, such as client behavior or
environmental conditions, is compared with the level observed in the
absence of the program or intervention. This can be done by comparing
Page 36 GAO-12-208G