Highlights of 2011 - Institute for Policy Research - Northwestern ...

looks at how to represent and combine the results of several
experiments, which can sometimes yield multiple measures
of the same outcome construct. The results will improve the
precision of estimates and suggest new ways to use the results
of randomized experiments in education.
About the Program
New Parameters for State Test Scores
IES is also sponsoring a project co-led by Hedges that seeks to
design new parameters for education experiments at the state,
local, school, and classroom levels. Many current education
experiments use designs that involve the random assignment
of entire pre-existing groups (e.g., classrooms and schools) to
treatments, but these groups are not themselves composed
at random. As a result, individuals in the same group tend to
be more alike than individuals in different groups, so results
obtained from single-district data might be too imprecise to
provide useful guidance. This project will decompose the
total variation of state achievement test scores to estimate
experiment design parameters for students in particular grades
in each state. The new parameters will take into account
achievement status and year-to-year improvement for a
particular grade, as well as demographic covariates. Designs will
also differ across different school contexts with a focus on lowperforming
schools, schools serving low-income populations, or
schools with large minority populations.
Pretreatment Effects in Political
Communication Experiments
Research on political communication effects has seen great
progress over the past 25 years. A key ingredient underlying
these advances is the increased usage of experiments that
demonstrate how communications influence opinions and
behaviors. But virtually none of these studies pay attention to
events that occur before the experiment, or “pretreatment
events.” Given that many, if not most, researchers design
experiments aimed at capturing “real world” political
communications, the likelihood of pretreatment contamination
is substantial. In a new working paper, IPR political scientist
James Druckman and IPR graduate research assistant Thomas
Leeper explore how and when the pretreatment environment
affects experimental outcomes. They present two studies—
one where they controlled the pretreatment environment and
one where it naturally occurred—to show how pretreatment
effects influence experimental outcomes, presenting the
first conclusive evidence of a pretreatment dynamic. More
importantly, they identify the conditions under which these
effects occur. When accounting for the pretreatment context,
they found that average experimental treatment effects might
miss important variations among subgroups. Furthermore, the
non-existence of experimental effects might stem from a large
number of individuals forming strong attitudes in response to
earlier communications prior to the experiment, making them
Larry Hedges, Chair
Most researchers and academics tend to stick
with the research methods they know best,
learned mainly in graduate school—even
though those methods might not represent
current best practices or the most appropriate
method. This is why statistician and education
researcher Larry Hedges, with the support of a
group of distinguished interdisciplinary scholars,
launched the Center for Improving Methods for
Quantitative Policy Research, or Q-Center, at
IPR. The work of Q-Center faculty often overlaps
with IPR’s Education Policy researchers.
more likely to reject subsequent contrary arguments. They
argue that, under certain conditions, attending to pretreatment
dynamics leads to a more accurate portrait of the mass public
and its political flexibility.
Party Heterogeneity in Candidates
IPR political scientist Georgia Kernell is examining the
conditions under which parties benefit from fielding more or
less heterogeneous candidate teams. While most spatial voting
models assume or imply that homogeneous candidate teams
offer parties the best prospect for winning elections—in reality,
candidates from the same political party often adopt divergent
policy positions. She reconciles theory and reality by identifying
a strategic rationale for political parties to recruit a diverse pool
of candidates. Kernell develops a spatial model in which two
parties each select a distribution of candidates to compete in
an upcoming election. The model demonstrates that parties
positioned close to the median voter should field a more
homogeneous set of candidates than parties with platforms
that are more distant. Kernell tests this prediction using data on
the policy positions of Democratic and Republican candidates
for congressional and state legislative elections since 1990.
In line with the model’s predictions, she finds that minority
parties—presumably more distant from the median voter—
are more heterogeneous than majority parties.
50 INSTITUTE FOR POLICY RESEARCH