GAO-12-208G, Designing Evaluations: 2012 Revision

Chapter 4: Designs for Assessing Program
Implementation and Effectiveness
Table 2: Common Designs for Implementation (or Process) Evaluations
Evaluation question Design
Is the program being implemented as intended? Compare program activities to statute and regulations, program
logic model, professional standards, or stakeholder expectations
Have any feasibility or management problems emerged? • Compare program performance to quality, cost or efficiency
expectations
• Assess variation in quality or performance across settings,
providers, or subgroups of recipients
Why is the program not (or no longer) achieving expected
outcomes?
Assessing Quality or the
Progress of Program
Implementation
Source GAO.
• Analyze program and external factors correlated with
variation in program outcomes
• Interview key informants about possible explanations
• Conduct indepth analysis of critical cases
Assessments of program implementation often compare program
performance—or what is—to a criterion established in advance—or what
should be. The evaluative criteria may be derived from the law,
regulations, a program logic model, administrative or professional
standards, research identifying the best practices of leading
organizations, or stakeholder expectations. Some criteria identify an
acceptable level of performance or performance standard by, for
example, defining authorized activities. In some areas, a program may
not be considered credible unless it meets well-established professional
standards. When criteria have no predetermined standard of acceptable
performance, the evaluator’s task is to measure the extent to which a
program meets its objectives. Measures of program performance may be
obtained from program records or may be specially collected for the
evaluation through interviews, observations, or systems testing. For
example,
• To assess the quality, objectivity, utility, and integrity of an agency’s
statistical program, an evaluator can compare its policies and
procedures for designing, collecting, processing, analyzing and
disseminating data with government guidelines for conducting
statistical surveys (OMB 2006).
• To evaluate the operational quality and efficiency of a program
providing financial assistance to individuals, an evaluator might
analyze administrative records that document the applications
received for program benefits and the actions taken on them.
Efficiency might be assessed by how promptly applications for
benefits were processed for a given level of staffing; quality might be
Page 32 GAO-12-208G