GAO-12-208G, Designing Evaluations: 2012 Revision

Assess the Relevance and
Quality of Available Data
Sources
Chapter 3: The Process of Selecting an
Evaluation Design
especially well or especially poorly. Findings supported by a number of
soundly designed and executed studies add strength to the knowledge
base exceeding that of any single study, especially when the findings are
consistent across studies that used different methods. If, however, the
studies produced inconsistent findings, systematic analysis of the
circumstances and methods used across a number of soundly designed
and executed studies may provide clues to explain variations in program
performance (GAO 1992b). For example, differences between
communities in how they staff or execute a program or in their client
populations may explain differences in their effectiveness.
A variety of statistical approaches have been proposed for statistically
cumulating the results of several studies. A widely used procedure for
answering questions about program impacts is “meta-analysis,” which is a
way of analyzing “effect sizes” across several studies. Effect size is a
measure of the difference in outcome between a treatment group and a
comparison group. (For more information, see Lipsey and Wilson 2000.)
Depending on the program and study question, potential sources for
evidence on the evaluation question include program administrative
records, grantee reports, performance monitoring data, surveys of
program participants, and existing surveys of the national population or
private or public facilities. In addition, the evaluator may choose to
conduct independent observations or interviews with public officials,
program participants, or persons or organizations doing business with
public agencies.
In selecting sources of evidence to answer the evaluation question, the
evaluator must assess whether these sources will provide evidence that
is both sufficient and appropriate to support findings and conclusions on
the evaluation question. Sufficiency refers to the quantity of evidence—
whether it is enough to persuade a knowledgeable person that the
findings are reasonable. Appropriateness refers to the relevance, validity,
and reliability of the evidence in supporting the evaluation objectives. The
level of effort required to ensure that computer-processed data (such as
agency records) are sufficiently reliable for use will depend on the extent
to which the data will be used to support findings and conclusions and the
level of risk or sensitivity associated with the study. (See GAO 2009 for
more detailed guidance on testing the reliability of computer-processed
data.)
Page 22 GAO-12-208G