In Search of Evidence

Chapter 6
simply be combined without closer consideration. A relatively straightforward
test, such as the so-called Forest plot in which the identified effect and the 95%
confidence interval of the individual studies are graphically displayed, can be
used to assess the statistical comparability (Lewis & Clarke, 2001). An example
of a Forrest plot is provided in Figure 1 2
Question 7: Has the meta-analysis been conducted correctly?
If the studies included are sufficiently comparable, statistical techniques can
be used to combine the outcomes of the individual studies. It is irrelevant
whether this relates to RCTs, observational research or non-comparative
research; as long as the outcome of the research has a point estimate and a
standard deviation can be calculated, the individual outcomes of the studies can
be combined.
One obvious way of combining the results of different studies is to count
the number of studies showing a positive effect of an intervention or success
factor and the number showing a negative effect and then compare the two
outcomes with each other. The validity of this method, termed ‘vote counting’,
has been acknowledged since the 1970s not to be very great because it does
not take account of essential factors such as sample size (small studies count
the same as large studies), research design (no distinction is made between
comparative research., surveys and case studies) and methodological quality
(poorly conducted studies or studies with a serious bias count the same as
qualitatively good studies) (Light & Smith, 1971). Even if these differences are
corrected by applying weighting factors, vote counting often delivers unreliable
outcomes (Hedges & Olkin, 1980; Hunter & Schmidt, 1990). Unfortunately vote
counting is still in common use, often implicitly, as revealed in phrases in the
conclusion such as “Most studies show….”.
Question 8: Are the conclusions valid and applicable?
The reader will have to estimate the validity of the systematic review on the
basis of the above aspects. The key questions to be considered are shown in
Table 3. If the answer to several of these questions is "No" or “Unclear”, this is
2
A few observations can be made from this example. Overall the 14 studies are heterogeneous, as is visible
by the dispersion of the CIs and de correlation coefficients. In addition, more than half of the studies (7) are
not significant, as the CIs cross or touch the line of zero effect.