Fall 2011 - Institute of Medical Science - University of Toronto

SURP RESEARCH FOCUS
Given the rigour of their design, are the results
of RCTs fool-proof? There are several
examples where the results of such trials did
not seem to agree with clinical reality. Rofecoxib
is a cyclooxygenase-2 inhibitor whose
use for the treatment of arthritis became
widespread after favourable results from
RCTs, only to be taken off the market a few
years later when it was found to increase the
risk of cardiovascular complications 6 . Reboxetine
was touted as an effective anti-depressant
until it was discovered that publication
of data had been highly selective – once the
complete body of data concerning drug efficacy
and safety were evaluated, it was found
that the drug was not only ineffective in the
treatment of depression, but harmful 7 .
In the above examples, the reported results
of RCTs did not correspond with reality because
of bias in trial conduct, analysis, or
publication 6,8 . Unfortunately, the degree to
which this sort of bias affects the published
results of any RCT is unknown.
Methods to evaluate bias in RCTs are important
for physicians who use results of RCTs
to guide treatment decisions. There is no
objective gold standard to evaluate bias because
it is difficult to measure and can only
be estimated 9 . An optimal assessment of bias
requires unrestricted access to both the procedures
used by the trial researchers and the
complete raw data, but such access is very
difficult to attain 10 . Nevertheless, there are
Adverse
Event
Not in results
table (NOT R)
In results table
(R)
Not in abstract
(NOT A)
In Abstract (A)
Not in discussion
(NOT D)
In discussion
(D)
Not in concluding
statement
(NOT C)
In concluding
statement (C)
certain criteria that can be used to estimate
the degree of bias in RCTs.
One criterion used to assess bias is the systematic
evaluation of the reporting of trial
endpoints. Endpoints are outcomes being
measured by the trial, which may include
overall survival, disease-free survival, quality
of life and response rate, among others. RCTs
are designed to recruit a predefined number
of people, and to determine if a statistically
significant difference in primary endpoints
exists 9 . This does not mean that significant
differences in other endpoints are not important,
but statistical tests applied to them are
subject to misinterpretation 8 . The evaluation
of secondary endpoints should therefore be
regarded as exploratory. If a publication does
not clearly indicate the results relating to the
primary endpoint of the trial and does not
describe the results of secondary endpoints
in its concluding statements, it is biased 8 .
Another possible criterion for the systematic
evaluation of bias is the reporting of adverse
events (AEs) associated with the experimental
treatment. We developed a method to
evaluate this bias, which employed a hierarchy
of AE reporting based on the sections of
a publication where AEs are most likely to be
read (Figure 1).
In each of 168 publications of RCTs evaluating
breast cancer treatment, every reported
moderate to severe AE that was statistically
Not in
discussion
(NOT D)
In discussion
(D)
Not in
discussion
(NOT D)
In discussion
(D)
NOT R
R + (NOT A) +
(NOT D)
R + (NOT A) + D
R + A + (NOT C) +
(NOT D)
R + A + (NOT C) +
D
R + A + C +
(NOT D)
R + A + C + D
Inadequate
reporting of
adverse events
Less adequate
reporting of
adverse events
Adequate
reporting of
adverse events
Figure 1. Hierarchy of adverse events (AE) reporting. One possible hierarchy scheme is shown, where
the top represents the least adequate reporting of a moderate to severe AE.
different between the experimental and control
arms received a score based on its position
in the hierarchy. This score was used to
cluster publications that had a similar reporting
of AEs. With a large enough sample of
publications, individual clusters could be defined
where each represents a certain degree
of bias. A survey querying oncologists about
where they most commonly see the reporting
of AEs in publications of RCTs has been designed
to test the validation of the hierarchy
in Figure 1. The results are pending.
There is substantial evidence that bias exists
in the conduct, analysis, and reporting
of RCTs 8-10 . A measure of the degree of this
bias would be of great help to those who must
decide how much to trust the results of these
RCTs, especially when deciding whether
to apply the results to patients. Although
no gold standard exists that can be used to
evaluate the degree of bias in a publication,
methods are being developed for the purpose
of estimating this bias with the hope of minimizing
its effect on clinical decision-making.
References
1. Concato J, Shah N, Horwitz RI. Randomized controlled
trials, observational studies, and the hierarchy of
research designs. NEJM 2000; 342(25): 1887-92.
2. FDA approval of new cancer treatment uses for marketed
drug and biological products. Food and Drug
Administration; c1998. Available from: http://www.fda.
gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/ucm071657.pdf
(accessed
August 2011)
3. Altman DG, Bland JM. How to randomize. BMJ 1999;
319: 703-4.
4. Schulz KF, Grimes DA. Allocation concealment in
randomised trials: defending against deciphering. Lancet
2002; 359: 614-8.
5. Schulz KF, Grimes DA. Blinding in randomised trials:
hiding who got what. Lancet 2002; 359: 696-700.
6. Roth-Cline MD. Clinical trials in the wake of Vioxx.
Circulation 2006; 113: 2253-59.
7. Eyding D, Lelgemann M, Grouven U, Harter M,
Kromp M, Kaiser T, Kerekes MF, Gerken M, Wieseler
B. Reboxetine for acute treatment of major depression:
systematic review and meta-analysis of published
and unpublished placebo and selective serotonin reuptake
inhibitor controlled trials. BMJ 2010; 341: c4737
doi:10.1136/bmj.c4737
8. Boutron I, Dutton S, Ravaud P, Altman DG. Reporting
and interpretation of randomized controlled trials
with statistically nonsignificant results for primary outcomes.
JAMA 2010; 303(20): 2058-64.
9. Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC,
Altman DG. Empirical evidence for selective reporting
of outcomes in randomized trials. JAMA 2004; 291(20):
2457-65.
10. Chan AW. Bias, spin, and misreporting: time for
full access to trial protocols and results. PLoS Medicine
2008; 5(11): 1533-35.
IMS MAGAZINE FALL 2011 PROSTATE CANCER | 30