What Is Evidence-Based Medicine?1 - Surgical Planning Laboratory

Special Report
Radiology Alliance for Health Services Research
What Is Evidence-Based Medicine? 1
Critical Thinking Skills Symposium
Kelly H. Zou, PhD, Julia R. Fielding, MD, Silvia Ondategui-Parra, MD, MPH, MSc
Rationale and Objectives. In this review article, we present the definition and useful concepts of evidence-based medicine
(EBM). The principles of EBM are provided and major steps of practicing EBM are described.
Materials and Methods. We emphasize the importance of the Cochrane Collaboration (see http://www.cochrane.org),
which initiated the research and practice in this area. Because it can be difficult to systematically access and review individual
research studies, it is often useful to focus on a critical overview of clinical trials by conducting a meta-analysis.
Results. Useful literature and resources related to meta-analysis are provided.
Conclusion. Statistical methods for evaluating radiologic diagnostic performances derived from meta-analysis are summarized,
with a special focus on summary outcomes measures.
Key Words. Evidence-based medicine; evidence-based radiology; meta-analysis; sensitivity; specificity; receiver operating
characteristic curve.
© AUR, 2004
Acad Radiol 2004; 11:127–133
1
From the Radiology Management Group, Department of Radiology,
Brigham and Women’s Hospital, Harvard Medical School, Boston, MA; Department
of Health Care Policy, Harvard Medical School, Boston, MA; Department
of Radiology, University of North Carolina at Chapel Hill, Chapel
Hill, NC; Department of Health Policy and Management, Harvard School of
Publish Health, Boston, MA. Received July 14, 2003; revision requested
August 28; received in revised form September 8; accepted September 9.
Partially supported by a research grant R03HS13234-01 from the National
Institutes of Health. Address correspondence to Kelly H. Zou e-mail:
zou@bwh.harvard.edu
© AUR, 2004
doi:10.1016/S1076-6332(03)00650-0
Evidence-based medicine (EBM), also called evidencebased
health care, was developed because of the awareness
of the limitations of traditional determinants. The
principles of EBM offer a useful solution to clinical problems
to acquire valid and current information for clinical
and policy decisions (1). EBM is defined as the process
of systematically finding, critically appraising, and using
contemporary research published in the medical literature
as a basis for making decisions regarding individual patient
care and health care policy. It is also a habit that is
recommended to all practicing radiologists so that they
use the best logical thinking techniques to derive treatment
plans (2–4).
A comprehensive review of EBM applied to radiology
was recently published (1). However, so far these developments
have received limited applications in radiology.
The link between EBM and evidence-based radiology is
aimed at integration of evaluative sciences and technology
assessment into clinical practice (2).
There are several approaches for EBM. It is recognized
that one of the best methods for reviewing a field of information
and to systematically access individual research
studies is to conduct a meta-analysis. It is important for
radiologists to realize that EBM offers solutions that can
be applied at many levels of professional involvement. A
unique feature of EBM is that it can be used readily by
practicing radiologists working at the effectiveness level:
performance in their own departments under ordinary
clinical practice (1).
127

ZOU ET AL Academic Radiology, Vol 11, No 2, February 2004
Table 1
Trends in Evidence-based Medicine From 1995 to 2002 Using
MEDLINE Search with Subject Heading Words “Evidencebased
Medicine”
Year Number of Publications Relative Frequency (%)
1995 20 0.2
1996 215 2.0
1997 678 6.2
1998 1075 9.8
1999 1611 14.7
2000 1889 17.2
2001 2573 23.4
2002 2919 26.6
Total 10980 100.0
First, in this article, we will focus on how to integrate
EBM techniques into daily clinical practice. We will then
review the basic steps required to perform a meta-analysis.
Finally, a summary and concluding remarks are provided.
EVIDENCE-BASED MEDICINE IN RADIOLOGY
RADIOLOGY
The number of articles published on EBM has seen a
dramatic increase. The trends of conducting EBM between
1995 and 2002, based on a MEDLINE search using
the subject heading “evidence-based medicine,” are
provided (Table 1). The major work of evidence-based
medicine is to draw conclusions from results collected
from literature, and alternatively, from narrative reviews
and data pooled from independent studies—often clinical
trials. The main methodologic approach is to perform a
systematic statistical analysis to extract, compare, and
combine the reported results from these studies to derive
quantifiable outcomes (2). Explicit criteria are necessary
to render an objective assessment of the study’s value.
Such criteria include inclusion and exclusion subject criteria
with assessment of bias, reasonableness of the clinical
intervention, appropriateness of the statistical evaluation,
and overall credibility when compared with a practitioner’s
medical knowledge.
There are four major steps to the practice of evidencebased
medicine (4), as described in the following sections.
Step 1. Formulate the Clinical Question
This first step is the single most important one, and
requires careful thought. It is best to formulate the clinical
question in the form of an exposure and patient outcome.
For example, “Does the use of contrast-enhanced
computed tomography (CT) scan of the chest (exposure/
possible harm) as a primary test for dyspnea in a 68-yearold
woman with a normal chest radiograph reduce her
risk of dying of pulmonary embolus?” The more detail
incorporated into the clinical question, the more specific
is the relevant literature review. In this example, the patient’s
age, symptoms, and chest radiograph findings are
included. Thus, in the literature review, one could disregard
articles describing young patients, asymptomatic patients,
and those patients with abnormal chest radiographs.
Step 2. Find the Evidence
Conventionally, radiologists may search for useful evidence
for clinical practice without applying appropriate or
scientific methods. For example, one may discuss a case
with a colleague or mentor, retrieve individual research
articles, and review quality and relevance-filtered publications.
The limitation in such practice is that the expert
opinion can be out of date and even incorrect, particularly
when confronted with a very rare disease or unusual manifestation
of a more common ailment. Alternatively, many
radiologists keep paper or electronic files of individual
research articles, usually divided by organ system. These
files, when critically reviewed and kept up to date, may
help understand the current status of imaging a particular
disease. Unfortunately, the clinical information from these
personal files typically does not represent that using a
population-based approach, which is relevant to the individual
case to be diagnosed or treated. Therefore, sophisticated
and robust methods are needed for systematically
combining evidence.
Computer databases such as MEDLINE and PUBMED
have shown to be invaluable in these cases. The Cochrane
Collaboration, based in the United Kingdom, recommends
performing meta-analyses rather than the ad hoc combination
of information present in the literature. Founded by
Dr. Archie Cochrane in 1972, the goal of the group is to
determine, in well-designed studies, which therapies are
effective and then to use health care resources to provide
these therapies to the population in an equitable and presumably
cost-effective manner. In this group, reviewers,
consumers, translators, and hand searchers submit work to
an editorial team assigned to a particular health care
project. Using predetermined eligibility criteria, studies
are selected to provide the largest dataset possible. A
meta-analysis is then performed on these studies. The
results of the analysis form the basis of a report that
128

Academic Radiology, Vol 11, No 2, February 2004
WHAT IS EVIDENCE-BASED MEDICINE?
makes recommendations on the usefulness of a particular
test or therapy. In the United States, for example, technology
assessment is reviewed by the Quality Interagency
Coordination Task Force based in Washington, DC.
Step 3. Critical Appraisal
When reviewing an individual research study or conducting
a meta-analysis, one must assess the work for
relevance and methodologic rigor. This is a critical step
for developing epidemiologic tools to assess the validity
and the quality of the evidence found in the literature.
There are different levels of evidence and grades for
recommendation. The medical literature may be classified
according to its quality level ranging from type (1), the
highest quality, to type (5) the lowest. Type (1) evidence
is from a systematic review, which includes at least one
randomized controlled trial and a summary of all included
studies. Examples of these studies include those published
by the Cochrane Collaboration, the National Health Service
Centre for Reviews and Dissemination, and the
Agency for Healthcare Research and Quality. The evidence
from such a review requires careful appraisal; if
well done, such evidence is powerful. Type (2) evidence
is from a properly designed randomized controlled trial of
appropriate size. Type (3) evidence is from a well-designed
intervention study without randomization. Evidence
in this category will only be included if no type (1)
or (2) evidence is available. A common research design is
the before-and-after study. Type (4) evidence is supplied
from a well-designed nonexperimental study (eg, cohort,
case-control or cross-sectional study and any study using
purely qualitative methods). Studies in this category will
only be included if no type (1), (2), or (3) evidence is
available. Economic analyses (cost-effectiveness studies)
are also classified as type (4) evidence. Type (5) evidence
consists of opinions of respected authorities based on
clinical evidence, descriptive studies, or reports of expert
consensus committees.
Step 4. Develop Solutions
Research results, even those that are carefully reviewed,
should not be used to determine patient treatment.
Instead, the practitioner should combine the information
gathered from literature reviews with his or her
clinical expertise and available external evidence. In most
cases, this evidence will consist of the patient’s history
and physical examination and laboratory test results. In
this way, the best diagnostic and therapeutic options
available can be matched to a specific patient’s condition.
Table 2
Trends in Meta-analysis Medicine From 1995 to 2002 Using
MEDLINE Search with Subject Heading Words “Metaanalysis”
Year Number of Publications Relative Frequency (%)
1995 274 8.6
1996 261 8.2
1997 331 10.4
1998 350 11.0
1999 397 12.5
2000 491 15.4
2001 544 17.1
2002 534 16.8
Total 3182 100.0
META-ANALYSIS
Definition and Trends
Meta-analysis is a quantitative method for combining
the results of independent studies, usually drawn from
published literature, and for synthesizing summaries and
conclusions, which may be used to evaluate therapeutic
effectiveness and plan new studies (5,6). It has increasingly
been used for evaluating and comparing diagnostic
performances, for example, of imaging modalities (7–9),
biopsy techniques (10), and vascular interventions (11).
The number of papers published on meta-analyses has
increased steadily in the past 10 years. See the trends of
conducting meta-analysis between 1995 and 2002 in the
results of a MEDLINE search using the subject heading
of “meta-analysis” (Table 2).
Advantages
For the purpose of critically evaluating a clinical hypothesis
based on published clinical trials, meta-analysis
is an efficient tool for summarizing the results in the literature
numerically. Meta-analysis allows for an objective
appraisal of the evidence, which may lead to resolution of
uncertainty and disagreement. It can reduce the probability
of false-negative results and thus prevent undue delays
in the introduction of effective treatments into clinical
practice. A priori hypotheses regarding treatment effects
in subgroups of patients may be tested. It may also explore
and sometimes explain the heterogeneity between
study results. Finally, the analysis may help guide the
design of future research. Specifically, the sample size
needed in future studies may be calculated more accurately.
Classic textbooks on meta-analysis have been writ-
129

ZOU ET AL Academic Radiology, Vol 11, No 2, February 2004
ten by several authors (12–15). In the late 1990s, the British
Journal of Medicine published a series of articles on
this topic (17–23).
Tools
There are several epidemiologic and statistical tools
required to scientifically synthesize and assemble literature
data in a meta-analysis: (1) a carefully considered
and detailed protocol should be written before beginning
the project; (2) an a priori definition of eligibility criteria
should be included, with a comprehensive search for such
studies as a central part of the work; (3) the results
should be graphed on a common scale to allow a visual
examination of the heterogeneity between studies; (4) an
appropriate statistical method should be chosen for combining
data; and (5) a thorough “sensitivity” analysis
should be performed to assess the robustness of combined
estimates using different assumptions and inclusion criteria.
In Table 3, we illustrate these five tools on a wellconducted
meta-analysis by Oei et al. (9). The authors
evaluated the diagnostic performance of magnetic resonance
imaging of the menisci and cruciate ligaments of
the knee and assessed the effect of study design characteristics
and magnetic field strength on diagnostic performance.
Based on 29 of the 120 retrieved articles, the authors
found that the performance of magnetic resonance
imaging differed according to lesion types and was influenced
by study designs. In addition, higher magnetic field
strength moderately improved diagnostic accuracy, with a
significant effect on the identification of anterior cruciate
ligament tears.
Sources of Biases
We have already defined the classification system of
quality of the literature under EBM. However, there are
several biases that may lead to an erroneous conclusion.
The most common types of bias include: (1) publication
bias: significant results are more likely to get published;
(2) language and citation bias: among published studies,
those with significant results are more likely to get published
in English, to be cited, and to be published repeatedly;
(3) database bias: in less developed countries, studies
with significant results may be more likely to get published
in a journal indexed in a literature database; and
(4) inclusion bias: criteria for including studies in a metaanalysis
may be influenced by knowledge of the results of
the set of potential studies. To minimize these possible
biases, one must search the world literature thoroughly
Table 3
A Step-by-Step Illustration of a Toolbox for Meta-analysis on
a Published Study on MRI of the Menisci and Cruciate
Ligaments
Step Tools Methods Applied
1. Detailed
protocol
2. Eligibility
criteria
3. Graphical
display
4. Statistical
methods
5. Sensitivity
analysis
Search engine: MEDLINE
Date: January 1991–December 2000
Purpose: Diagnostic performance of MRI
of knee lesions
Terms: Magnetic resonance imaging, knee,
meniscus, cruciate ligament, arthroscopy
Extractors: Two independent readers, with
third reader assessing discrepancies
Study characteristics: Publication year,
country, setting, patient characteristics,
aspects of study design, verification
bias, characteristics of MRI
Likelihood for inclusion: Yes, possible, no
(1) Inclusion criteria:
Language: English
Diagnosis: MRI of lesions of medial or
lateral meniscus, ACL, or PCL
Sample size: At least 30 subjects
Gold standard: Arthroscopy
Measurement: Magnetic field strength
Threshold: Positive criteria for MRI
Outcome data: Absolute numbers of TP,
FN, TN, and FP
(2) Exclusion:
Patient population: Consists of infants or
adolescents
Study objective: MRI for postoperative
evaluation
Study design: Case-control
Ligaments: Only the medial and lateral
meniscus combined
Measurement: Various magnetic field
strengths
Outcome data: Only the diagnostic value
of specific features and indirect signs of
knee lesions at MRI
Funnel plot of log odds ratio
Summary ROC plot
Pooled weighted analysis of sensitivity and
specificity
Random effects model
Summary ROC analysis per lesion and for
all lesions
Delete-one Jackknife method
Note.—MRI magnetic resonance imaging; ACL anterior
cruciate ligament; PCL posterior cruciate ligament; TP true
positive; FN false negative; TN true negative; FP false
positive; ROC receiver operating characteristic.
130

Academic Radiology, Vol 11, No 2, February 2004
WHAT IS EVIDENCE-BASED MEDICINE?
Table 4
Summary Measures Commonly Used for Conducting Meta-analysis in Radiology
Summary Measure
Sensitivity (true positive rate)
Specificity (true negative rate)
Positive predictive value
Negative predictive value
Likelihood ratio
Odds ratio
Relative risk
Receiver operating curve
Definition
The proportion of subjects with disease who have a positive test.
The proportion of subjects without disease who have a negative test
The proportion of test positive subjects who truly have disease.
The proportion of test negative subjects who truly do not have disease.
The probability that a subject with disease would have a particular test result divided by the probability
that a subject without the disease would have that result.
The probability of the disease occurring divided by the probability that it doesn’t occur.
The probability of the disease in the risk group divided by the probability of the disease in the control
group.
A plot of (1-specificity, sensitivity) at all possible decision threshold.
Table 5
Absolute Diagnostic Rates in the Computed Tomography of Metastasis in Lung Cancer(35)
Study
Sample
Size (n)
True
Positive
(TP)
False
Negative
(FN)
False
Positive
(FP)
True
Negative
(TN) Sensitivity Specificity
1 50 11 7 6 26 0.611 0.813
2* 35 2 5 15 13 0.287 0.464
3 51 15 2 2 32 0.833 0.941
4 22 4 1 4 13 0.800 0.765
5 50 22 21 1 6 0.512 0.857
6 42 17 1 9 15 0.944 0.625
7 94 29 10 1 54 0.744 0.982
8 41 8 6 4 23 0.571 0.852
9 50 7 6 12 25 0.538 0.676
10 49 20 1 10 18 0.952 0.643
11 48 19 1 9 19 0.950 0.679
12 97 18 6 8 65 0.750 0.890
13 41 18 1 9 13 0.947 0.591
14 75 17 3 6 49 0.850 0.891
Note—*Data from Study 2 were not included in the analysis because of inhomogeneity. Sensitivity TP/(TP FN) and Specificity
TN/(FP TN)
and use eligibility criteria stringently (24–26). A tutorial
article addressing the statistical methods for meta-analysis
including reduction of bias was published in 1999 by
Normand (27).
Diagnostic Imaging: A Receiver Operating
Characteristic Curve
There are several summary outcomes derived from
meta-analyses useful for imaging research (28–30). These
include sensitivity, specificity, predictive value, likelihood
ratio, odds ratio, relative risk, and summary receiver operating
characteristic (SROC) curve (Table 4). SROC methods
are designed to show the accuracy of a specific test at
predetermined sensitivity and specificity levels. There are
several methods to derive such a curve. Kardaun and
Kardaun (31) used a bivariate normal maximum-likelihood
method. Littenberg et al. (32) employed a logit difference–sum
regression model, in which the logit-transformed
true-positive fraction and false-positive fraction
have a linear relationship; logit(p) ln{p/(1p)}. Recently,
the logit difference–sum method was validated via
statistical simulations (33). A more complicated approach
using a latent-scale logistic regression analysis model was
developed by Rutter and Gatsonis (34).
As a simple example for derivation of the summary
ROC curve, Table 5 provides the subset data from 14
131

ZOU ET AL Academic Radiology, Vol 11, No 2, February 2004
Figure 1. Summary Receiver Operating Curves of Diagnostic Rates in the Computed
Tomography of Metastasis in Lung Cancer by Two Estimation Methods.
studies found by Inouye and Sox (35) and analyzed by
both Kardaun and Kardaun and by Litternberg et al. The
intention of this meta-analysis was to evaluate the diagnostic
accuracy of CT scans for detecting metastases in
patients with non–small-cell lung cancer. Summary ROC
curves were constructed based on 13 studies. Data from
study 2 were omitted because the sensitivity and specificity
were less than 0.5, which was not homogeneous with
remaining data. The two analysis models yielded the summary
ROC curves, both displayed in Figure 1. The logit
difference–sum model gave the estimated regression equation
logit(TP) 3.26 1.12 logit(FP), whereas the
bivariate normal maximum-likelihood model gave the
estimated equation logit(TP) 3.71 1.64 logit(FP).
The large area under the ROC curves showed that CT
was accurate in diagnosis of metastatic lung nodes.
SUMMARY
In this brief review article, we present the concepts
and steps for EBM and meta-analysis, with a focus on
evidence-based radiology. Careful scientific methodology
is called for to minimize biases and to systematically synthesize
medical literature. The application of EBM principles
to diagnostic imaging can facilitate the interpretation
of imaging studies and create a well-conducted radiologic
evaluation.
REFERENCES
1. The Evidence-Based Radiology Working Group. Evidence-based
radiology: a new approach to the practice of radiology. Radiology
2001; 220:566–575.
2. Eisenberg JM. Then lesions for evidence-based technology assessment.
JAMA 1999; 282:1865–1869.
3. Hoffrage U, Lindsey S, Hertwig R, et al. Medicine communicating statistical
information. Science 2000; 290:2261–2262.
4. Sackett DL, Straus SE, Richardson S, et al: Evidence-based medicinehow
to practice and teach EBM. 2nd ed. Edinburgh, Scotland:
Churchill Livingstone, 2000.
5. Jaeschke R, Guyatt G, Sackett DLfor the Evidence-Based Medicine
Working Group. Users’ guides to the medical literature, III. How to
use an article about a diagnostic test. B. What are the results and
will they help me in caring for my patients? JAMA 1994; 271:703–
707.
6. Langlotz CP, Sonnad SS. Meta-analysis of diagnostic procedures: A
brief overview. Academic Radiology 1998; 5(Suppl. 2):S269–S273.
132

Academic Radiology, Vol 11, No 2, February 2004
WHAT IS EVIDENCE-BASED MEDICINE?
7. Visser K, Hunink MG. Peripheral arterial disease: A gadolinium-enhanced
MR angiography versus color-guided duplex US—meta-analysis.
Radiology 2000; 216:67–77.
8. Kinkel K, Lu Y, Both M, et al. Detection of hepatic metastases from
cancers of the gastrointestinal tract by using noninvasive imaging
methods (US, CT, MR imaging, PET): A meta-analysis. Radiology
2002; 224:748–756.
9. Oei EHG, Nikken JJ, Verstignen ACM, et al. MR imaging of the menisci
and cruciate ligaments: A systematic review. Radiology 2003;
226:837–848.
10. Samson DJ, Flamm CR, Pisano ED, et al. Should FDG PET be used to
decide whether a patient with an abnormal mammogram or breast
finding at physical examination should undergo biopsy? Acad Radiol
2002; 9:773–783.
11. MacEneaney PM, Malone DE. Applying “evidence-based medicine”
theory to interventional radiology. II. A spreadsheet for swift assessment
of procedural benefit and harm. Clin Radiol 2000; 55:938–945.
12. Hedges LV, Olkin I. Statistical methods for meta-analysis. New York:
Academic Press, 1985.
13. Eddy DM, Hasselblad V, Shachter R. Meta-analysis by the confidence
profile method. The statistical synthesis of evidence. Boston: Academic
Press, 1992.
14. Petitti DB. Meta-analysis, decision analysis, and cost-effectiveness
analysis. Methods for quantitative synthesis in medicine. New York:
Oxford University Press, 1994.
15. Cooper H, Hedges LV. The handbook of research synthesis. New
York: Russell Sage Foundation, 1994.
16. Rosenberg W, Donald A. Evidence based medicine: An approach to
clinical problem-solving. BMJ 1995; 310:1122–1126.
17. Sackett DL, Rosenberg WM, Gray JA, et al. Evidence based medicine:
What it is and what it isn’t. BMJ 1996; 312:71–72.
18. Egger M, Smith DG. Meta-analysis: Potential and promise. BMJ 1997;
315:1371–1374.
19. Egger M, Davey Smith G. Meta-analysis: Principles and procedure.
BMJ 1997; 315:1533–1537.
20. Smith DG, Egger M, Phillips AN. Meta-analysis: Beyond the grand
mean. BMJ 1997; 315:1610–1614.
21. Egger M, Smith DG. Meta-analysis: Bias in location and selection
studies. BMJ 1998; 316:61–66.
22. Egger M, Schneider M, Smith DG. Spurious precision? Meta-analysis
of observational studies. BMJ 1998; 316:140–144.
23. Smith DG, Egger M. Meta-analysis: Unresolved issues and future developments.
BMJ 1998; 316:221–225.
24. Lyengar S, Greenhouse J. Selection models and the file-drawer problem.
Stat Sci 1988; 3:109–135.
25. Dear KB, Begg CB. An approach for assessing publication bias prior
to performing a meta-analysis. Stat Sci 1992; 7:237–245.
26. Hedges LV. Modeling publication selection effects in meta-analysis.
Stat Sci 1992; 7:246–255.
27. Normand S-LT. Meta-analysis: Formulating, evaluating, combining,
and reporting. Stat Med 1999; 18:321–359.
28. Rockette HE, Gur D, Campbell WL, et al. Use of meta-analysis in the
evaluation of imaging systems. Acad Radiol 1994; 1:63–69.
29. Walter SD, Jadad AR. Meta-analysis of screening data: A survey of
the literature. Stat Med 1999; 18:3409–3424.
30. Irwig L, Macaskill P, Glasziou P, et al. Meta-analytic methods for diagnostic
test accuracy. J Clin Epidemiol 1995; 48:119–130.
31. Kardaun JWPF, Kardaun OJWF. Comparative diagnostic performance
of three radiological procedures for detection of lumbar disk herniation.
Meth Info Med 1990; 29:12–22.
32. Moses LE, Shapiro D, Littenberg B. Combining independent studies of
a diagnostic test into a summary ROC curve: Data-analytic approaches
and some additional considerations. Stat Med 1993;
12:1293–1316.
33. Mitchell MD. Validation of the summary ROC for diagnostic test metaanalysis:
A monte-carlo simulation. Acad Radiol 2003; 10:25–31.
34. Rutter CM, Gatsonis CA. Regression methods for meta-analysis of
diagnostic test data. Acad Radiol 1995; 2(Suppl. 1):S48–S56.
35. Inouye SK, Sox HC. Standard and computed tomography in the evaluation
of neoplasms of the chest. Ann Int Med 1986; 105:906–924.
133