IMS Magazine - Summer 2012 edition in PDF format - Institute of ...

VIEWPOINT
survival of the fittest author, and away from
disseminating authentic results. Scientists
are increasingly being evaluated using metrics
such as the h-index, which is based on
number of publications and number of citations
per publication 5 . Assessing a scientist’s
performance using such metrics creates undeniable
desire, need, and pressure to publish
often, and in highly cited journals. Countries
such as China, South Korea, and Turkey offer
cash incentives to encourage local scientists
to submit their manuscripts to high-impact
journals 6 . Despite the low probability of success,
the high volume of submissions overwhelms
reviewers and congests the publishing
pipeline. Given these circumstances, how
can the literature be trusted, and how can
scientific progress be made?
The soaring retraction rate observed
in high-impact journals may
be an accelerated manifestation of
the decline effect. Because highimpact
journals publish studies
with novel and dramatic results,
those results are more likely to be
overstated, and are less likely to be
reproducible. The lack of reproducibility
is unsurprising considering
the tweaking, selecting, and beautifying
of data that oftentimes precedes
publication.
First reported in the 1930s, an established
barrier to scientific advancement is the decline
effect, which is observed in the literature
as scientifically discovered effects that
diminish over time. Schooler suggests that
“if early results are more likely to be reported
when errors combine to magnify the apparent
effect, then published studies will show
systematic bias towards initially exaggerated
findings, which are subsequently statistically
self-corrected” 7 . The soaring retraction rate
observed in high-impact journals may be an
accelerated manifestation of the decline effect.
Because high-impact journals publish
studies with novel and dramatic results, those
results are more likely to be overstated, and
are less likely to be reproducible. The lack of
reproducibility is unsurprising considering
the tweaking, selecting, and beautifying of
data that oftentimes precedes publication.
Detrimentally, if the studies in high-impact
journals are reaching the widest audiences
and are conveying less-than-accurate results,
the scientific field is being misguided.
The argument can be made that science is an
imperfect field, with enormous variability. In
biomedical research, there are innumerable
known and unknown variables that influence
experimental outcomes, some as significant
as the strain of mouse used, some as innocuous
as the time of day an experiment is conducted.
Every experiment contains outliers,
inconsistent replicates, and unexpected findings,
but these are rarely reported, and perhaps
that is the bigger issue. As Dr. Karen Davis,
Associate Director of the IMS and Editor
of Pain, previously communicated to the IMS
Magazine (“Positive Pressure,” Fall 2011), an
incomplete or incorrect scientific record can
lead to propagation of unfounded ideas, unnecessary
replication of experiments, clinical
translation of harmful therapies, or delayed
development of alternate hypotheses. Once
flawed ideas are published, they can never
truly be retracted. Budd et al. report that
even after an article is retracted, it continues
to be cited, without acknowledgement of the
retraction 3 . Understandably, once digital versions
of articles are downloaded, researchers
are unlikely to consult the original source
again, and are therefore unlikely to be aware
of corrections.
What is needed is a more complete account
of scientific findings, be they negative or
positive, consistent or inconsistent, surprising
or expected. During the IMS Scientific
Day 2012 Bernard Langer Lecture in Health
Sciences, Dr. Thomas R. Insel, MD, Director
of National Institute of Mental Health, posited
that if experiments are soundly designed
and flawlessly executed, then those results
should be disseminated, regardless of what
they are. Furthermore, Schooler believes “we
need a better record of unpublished research
before we can know how well the current
scientific process, based on peer review and
experimental replication, succeeds in distinguishing
grounded truth from unwarranted
fallacy” 7 . He recognizes the difficulties in
implementation, but suggests an open-access
database of research methods, which allows
scientists to log their hypotheses and methodologies
prior to experiments, and their
published and unpublished results afterwards
7 .
Schooler believes “we need a better
record of unpublished research
before we can know how well the
current scientific process, based on
peer review and experimental replication,
succeeds in distinguishing
grounded truth from unwarranted
fallacy.”
To purify the publication pool, the current
publishing paradigm should be restructured
to diffuse the pressure experienced by researchers.
Among other strategies, graduate
students should be given avenues to publish
their negative results, and senior scientists
should be evaluated on metrics other than
their publishing records. To purify the publication
pool, more focus should be placed
on honest accounting of data in its entirety,
and less focus should be placed on polishing
of data for high-impact journals. Researchers
should be mindful of the permanence of
published results, and wary that embellished
or modified data can significantly hinder
scientific progress. To alleviate current publishing
pressures is to purify the publication
pool, and move science forward.
Disclaimer: The opinions expressed by the
author(s) are in no way affiliated with the Institute
of Medical Science or the University of Toronto.
Comments are welcome at theimsmagazine@
gmail.com.
References
1. Fang FC, Casadevall A. Retracted science and the retraction
index. Infect Immun. 2011;79(10):3855-3859.
2. Zimmer C. A Sharp Rise in Retractions Prompts Calls
for Reform. The New York Times. 2012 Apr 17;Sect. D-1.
3. Van Noorden R. Science publishing: The trouble with
retractions. Nature. 2011;478(7367):26-28.
4. Fanelli D. “Positive” results increase down the Hierarchy
of the Sciences. PLoS One. 2010;5(4):e10068.
5. Hirsch JE. An index to quantify an individual’s scientific
research output. Proc Natl Acad Sci U S A.
2005;102(46):16569-16572.
6. Stephan P. Research efficiency: Perverse incentives.
Nature. 2012;484(7392):29-31.
7. Schooler J. Unpublished results hide the decline effect.
Nature. 2011;470(7335):437.
IMS MAGAZINE SUMMER 2012 GENOMIC MEDICINE | 32