GAO-12-208G, Designing Evaluations: 2012 Revision

Ensure That Questions Are
Appropriate to the
Program’s Stage of
Maturity
Chapter 2: Defining the Evaluation’s Scope
the most important questions that can be feasibly addressed. The
evaluator can use the program’s logic model to organize the discussion
systematically to learn whether the requester’s concerns focus on how
the program is operating or whether it is achieving its intended results or
producing unintended effects (either positive or negative). It is also
important to ensure that the evaluation question is well-matched to the
program’s purpose and strategies. For example, if a program is targeted
to meet the housing needs of low-income residents, then it would be
inappropriate to judge its effectiveness by whether the housing needs of
all residents were met.
It is important to learn whether the requester has a specific set of criteria
or expectations in mind to judge the program against and whether
questions pertain to the entire program or just certain components. A
general request to “assess a program’s effectiveness” should be clarified
and rephrased as a more specific question that ensures a common
understanding of the program’s desired outcomes, such as, “Has the
program led to increased access to health care for low-income
residents?” or “Has it led to lower incidence of health problems for those
residents?” It is also important to distinguish questions about the overall
effectiveness of a nationwide program from those limited to a few sites
that warrant study because they are especially promising or problematic.
The difference is extremely important for evaluation scope and design,
and attention to the difference allows the evaluator to help make the study
useful to the requester.
Although the feasibility of the evaluation questions will continue to be
assessed during the design phase, an evaluator should gain agreement
on these questions before completing the design of the evaluation. If
program stakeholders perceive the questions as objective and reflecting
their key concerns, they will be more likely to find the evaluation results
credible and persuasive and act on them.
Different questions tend to be asked at different stages of program
maturity and often reflect whether the purpose of the study is to assist
program improvement or provide accountability. Three types of evaluation
are defined by whether the focus is on the program’s operations or
outcomes, or on the program’s causal link to the observed results. Of
course, a single study may use different approaches to address multiple
questions. (See table 1.)
Page 14 GAO-12-208G