annual report 2011 - Office for Research - Northwestern University

Charles Manski
Weinberg College of Arts and Sciences and
Institute for Policy Research
Debating the Impact of Public Policy
Both politicians and journalists should be deeply
skeptical of analysts in academia, think tanks, and
government agencies who seem unwavering in their
policy predictions. Writing in the August 2011 issue of
the Economic Journal, Charles Manski, economics, urged
policy analysts to improve the credibility of their studies
and better inform policy formation by being upfront
about their predictions’ uncertainty.
Manski exposes various practices of incredible certitude
in the hope that their identification will lead to more
useful policy analysis. He starts with the problem of
“conventional certitude”—the prediction equivalent of
conventional wisdom or a conclusion that is generally
accepted as true but might not be.
An example is the “scores” or point predictions
produced by the US Congressional Budget Office (CBO)
to assess the budgetary impact of legislation 10 years
out. CBO scores are often used to stake out legislative
positions even though they are based on strong and
unsubstantiated assumptions.
Scoring the recent US healthcare reform legislation, CBO
director Douglas Elmendorf reported to Congress that its
enactment “would produce a net reduction of changes
in federal deficits of $138 billion over the 2010–19 period
as a result of changes in direct spending and revenue.”
Manski points out that anyone seriously contemplating
the many changes to federal law embodied in this
legislation should recognize that the prediction of a
$138 billion deficit reduction is nothing more than a very
rough estimate. Yet Elmendorf expressed no uncertainty
about the figure, subsequently reported by the media
without questioning the reliability of the number.
Douglas Holtz-Eakin, a former CBO director, made a very
different prediction. In an op-ed in The New York Times,
he predicted that the legislation would increase deficits
by $562 billion rather than decrease them by $138
billion. Thus, the former and present directors of the CBO
made predictions that differed by $700 billion, yet each
presented his prediction as certain.
In the end, Manski suggests that policy makers, the
media, and the public should be deeply skeptical of
those who appear unwavering in their policy predictions.
He recommends recognizing ambiguity rather than
ignoring it.
Patricia Reese
Excellence in Research | Annual Report 2011 43