IPR - Institute for Policy Research - Northwestern University

Quantitative Methods for Policy research
< Quasi-Experimental Methods and Designs
Thomas D. Cook, Joan and Sarepta Harrison Chair
in Ethics and Justice, continues his work on quasiexperimental
alternatives to random assignment,
focusing mostly on two methods: regressiondiscontinuity
designs and propensity score matching.
Vivian Wong and Thomas Cook collaborate on
examining regression-discontinuity designs
and methodology.
Cook, a social psychologist, and IPR graduate research
assistant Vivian Wong recently published a paper
reviewing whether regression-discontinuity studies
reproduce the results of randomized experiments conducted
on the same topic. They enumerate the general
conditions necessary for a strong test of correspondence
in results when an experiment is used to validate any
nonexperimental method. They identify three studies
where regression discontinuity and experimental results
with overlapping samples were explicitly contrasted. By
the criteria of both effect sizes and statistical significance
patterns, they then show that each study produced similar
results. This correspondence is what theory predicts.
To achieve it in the complex social settings in which
these within-study comparisons were carried out, however,
suggests that regression discontinuity results might
be more generally robust than some critics contend.
Cook and IPR postdoctoral fellow Manyee Wong
are investigating further potential for regressiondiscontinuity
designs to see if such designs can handle
multiple variables in general. They are using recent data
from No Child Left Behind (NCLB), a program that
uses multiple criteria to select children for remedial
education services. In this type of analysis, the estimand
is no longer a single point; instead, it becomes an
intersection point of several independent variables on a
multidimensional plane.
In conjunction with IPR visiting scholar Peter Steiner,
Cook is also examining the use of matching as an
analytic substitute for randomization. Cook and Steiner
demonstrate why propensity score methods—coupled
with observational data—can be used to recreate the
results of a randomized experiment. They find that the
key to reducing bias when faced with the unreliability of
predictors is to select the “right” covariates and to make
sure those covariates are measured well. In future work,
they hope to develop better indicators for which covariates
are the “right” ones in various research contexts.
< Handbook of Meta-Analysis
Larry Hedges, Board of Trustees Professor of Statistics
and Social Policy, has finished editing the second
edition of The Handbook of Research Synthesis and Meta-
Analysis (Russell Sage Foundation, 2009) with Harris
Cooper of Duke University and Jeff Valentine of the
University of Louisville. Updating the first edition,
which became the most-cited reference book in the
field, the new edition incorporates state-of-the-art
techniques from all quantitative synthesis traditions.
Distilling a vast technical literature and many informal
sources, the handbook provides a portfolio of the most
effective solutions to the problems of quantitative data
integration. Among the statistical issues addressed by
the authors are the synthesis of non-independent data
sets, fixed and random effects methods, the performance
of sensitivity analyses and model assessments, and the
problem of missing data. In response to the increased
use of research synthesis in formulating public policy,
the second edition includes several new chapters.
One is on the strengths and limitations of research
synthesis in policy debates and decisions. Another looks
at computing effect sizes and standard errors from
clustered data, such as schools or clinics.
< Randomization in Education Research
Many researchers believe that randomized
experimentation is usually the best methodology for
investigating issues in education. However, it is not
42