Highlights of 2011 - Institute for Policy Research - Northwestern ...

tended to focus on differences in averages across groups.
This is consistent with most demographic research, which
has focused on rates rather than totals. Total numbers of
people with certain types of human capital are important for
U.S. competitiveness, however. Thus, Spencer is developing a
new model that allows for aging and retirement, international
movement, and potential policy effects of improved incentives
for attracting and training students. Having a framework for
systematically organizing information about human capital
could help U.S. policymakers both in tracking progress and
in developing strategies to increase particular kinds of human
capital. Spencer also hopes the statistics will be useful in
discussions about the future of U.S. higher education, and, by
extension, K–12 and even preschool education.
Reliable Covariate Measurement
The effect of unreliability of measurement on propensity score
adjusted treatment effects is a topic that has largely been
unexamined. A study published in the Journal of Educational
and Behavioral Statistics by IPR social psychologist Thomas D.
Cook and his colleagues Peter Steiner of the University of
Wisconsin–Madison and William Shadish of the University of
California, Merced presents results from their work simulating
different degrees of unreliability in the multiple covariates that
were used to estimate a propensity score. The simulation uses
the same data as two prior studies where the researchers
showed that a propensity score formed from many covariates
demonstrably reduced selection bias. They also identified
the subsets of covariates from the larger set that were most
effective for bias reduction. Adding different degrees of random
error to these covariates in a simulation, the researchers
demonstrate that unreliability of measurement can degrade the
ability of propensity scores to reduce bias. Specifically, increases
in reliability only promote bias reduction if the covariates
are effective in reducing bias to begin with. They found that
increasing or decreasing the reliability of covariates that do not
effectively reduce selection bias makes no difference at all.
Random and Cutoff-Based Assignment
Cook and IPR postdoctoral fellow Vivian Wong co-authored a
study in Psychological Methods reviewing past studies comparing
randomized experiments to regression discontinuity designs,
which mostly found similar results—but with some significant
exceptions. The authors argue that these exceptions might be
due to potential confounds of study characteristics with assignment
method or with failure to estimate the same parameter
over methods. In their study, they correct the problems by randomly
assigning 588 participants to be in a randomized experiment
or a regression discontinuity design in which they are otherwise
treated identically, comparing results estimating both the
same and different parameters. Analysis includes parametric,
semiparametric, and nonparametric methods of modeling
nonlinearities. Results suggest that estimates from regression
discontinuity designs approximate the results of randomized
experiments reasonably well but also raise the issue of what
constitutes agreement between the two estimates.
Accounting for Missing Survey Data
Missing data are prevalent in social science and health studies,
both in the form of attrition—in which responses “drop out”
of the data set after a certain point—and in nonmonotone
patterns of intermittently missing values. Yet even within these
patterns, not all missing data can be treated equally; certain
trends in missing data might indicate wider trends that should
be taken into account when forming conclusions about the data
set as a whole. In an article published in Biometrics, marketing
professor and IPR associate Yi Qian, with Hua Yun Chen and
Hui Xie of the University of Illinois at Chicago investigate the
use of a generalized additive missing data model that, contrary
to the existing literature, does not assume a restricted linear
relationship between missing data and the potentially missing
outcome. Using a bone fracture data set, they conduct an
extensive simulation study. Their simulation shows that the
proposed method helps reduce bias that might arise from the
misspecification of the functional forms of predictors in the
missing data model.
Learning Interventions Institute
Hedges co-led the 2011 American Society for Microbiology
(ASM)/National Institute of General Medical Sciences
Learning Interventions Institute on “Understanding Research
Techniques to Study Student Interventions,” held January 10–
13 in Washington, D.C. The institute aims to introduce new
behavioral and social science research methods that can be
used to understand factors affecting student interest, motivation,
and preparedness for research careers in science and medicine.
The program used an intensive “learn, apply, and share” process
of lectures and discussions, followed by small group work. All
elements of the process are focused on how research can be
used to learn what efforts drive and support academic success
and commitment by students who are studying in science,
technology, engineering, and mathematics (STEM) fields.
Evaluating Fellowship Programs
Evaluating the quality of researchers is a key component of
any strategy to improve the overall quality of research output.
Hedges and IPR research associate Evelyn Asch were part of
the Spencer Foundation’s full-scale review of its two highly
prestigious fellowship programs, designed to determine the
programs’ effectiveness in helping fellows become stronger
researchers than they would be otherwise. Hedges and Asch,
along with graduate student Jennifer Hanis, completed evaluations
of the Spencer Postdoctoral Fellowship program and
52 INSTITUTE FOR POLICY RESEARCH