GAO-12-208G, Designing Evaluations: 2012 Revision

Criteria for a Good
Design
Chapter 3: The Process of Selecting an
Evaluation Design
adequately addressed. Evaluators must be explicit about the limitations of
the study. They should ask, How conclusive is the study likely to be, given
the design? How detailed are the data collection and data analysis plans?
What trade-offs were made in developing these plans?
GAO and other organizations have developed guidelines or standards to
help ensure the quality, credibility, and usefulness of evaluations. (See
appendix I and the guidance in GAO’s design matrix, figure 2, as an
example.) Some standards pertain specifically to the evaluator’s
organization (for example, whether a government auditor is independent),
the planning process (for example, whether stakeholders were
consulted), or reporting (for example, documenting assumptions and
procedures). While the underlying principles substantially overlap, the
evaluator will need to determine the relevance of each guideline to the
evaluator’s organizational affiliation and their specific evaluation’s scope
and purpose.
Strong evaluations employ methods of analysis that are appropriate to the
question; support the answer with sufficient and appropriate evidence;
document the assumptions, procedures, and modes of analysis; and rule
out competing explanations. Strong studies present questions clearly,
address them appropriately, and draw inferences commensurate with the
power of the design and the availability, validity, and reliability of the data.
Thus, a good evaluation design should
• be appropriate for the evaluation questions and context. The design
should address all key questions, clearly state any limitations in
scope, and be appropriate to the nature and significance of the
program or issue. For example, evaluations should not attempt to
measure outcomes before a program has been in place long enough
to be able to produce them.
• adequately address the evaluation question. The strength of the
design should match the precision, completeness, and
conclusiveness of the information needed to answer the questions
and meet the client’s needs. Criteria and measures should be
narrowly tailored, and comparisons should be selected to support
valid conclusions and rule out alternative explanations.
• fit available time and resources. Time and cost are constraints that
shape the scope of the evaluation questions and the range of
Page 28 GAO-12-208G