GAO-12-208G, Designing Evaluations: 2012 Revision

Comparison Group Quasiexperiments
Chapter 4: Designs for Assessing Program
Implementation and Effectiveness
over both program implementation and external factors that may influence
program results. In fact, enforcing strict adherence to program protocols
in order to strengthen conclusions about program effects may actually
limit the ability to generalize those conclusions to less perfect, but more
typical program operations.
Ideally, randomized experiments in medicine are conducted as doubleblind
studies, in which neither the subjects nor the researchers know who
is receiving the experimental treatment. However, double-blind studies in
social science are uncommon, making it hard sometimes to distinguish
the effects of a new program from the effects of introducing any novelty
into the classroom or workplace. Moreover, program staff may jeopardize
the random assignment process by exercising their own judgment in
recruiting and enrolling participants. Because of the critical importance of
the comparison groups’ equivalence for drawing conclusions about
program effects, it is important to check the effectiveness of random
assignment by comparing the groups’ equivalence on key characteristics
before program exposure.
Because of the difficulties in establishing a random process for assigning
units of study to a program, as well as the opportunity provided when only
a portion of the targeted population is exposed to the program, many
impact evaluations employ a quasi-experimental comparison group
design instead. This design also uses a treatment group and one or more
comparison groups; however, unlike the groups in the true experiment,
membership in these groups is not randomly assigned. Because the
groups were not formed through a random process, they may differ with
regard to other factors that affect their outcomes. Thus, it is usually not
possible to infer that the “raw” difference in outcomes between the groups
has been caused by the treatment. Instead, statistical adjustments such
as analysis of covariance should be applied to the raw difference to
compensate for any initial lack of equivalence between the groups.
Comparison groups may be formed from the pool of applicants who
exceed the number of program slots in a given locale or from similar
populations in other places, such as neighborhoods or cities, not served
by the program. Drawing on the research literature to identify the key
factors known to influence the desired outcomes will aid in forming
treatment and comparison groups that are as similar as possible, thus
strengthening the analyses’ conclusions. When the treatment group is
made up of volunteers, it is particularly important to address the potential
for “selection bias”—that is, that volunteers or those chosen to participate
Page 42 GAO-12-208G