Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27non-lymphatic leukaemia, the risk ratio in RCTswas 24 compared with 3.7 in non-randomisedstudies (comparison 23 in Table 3).The impact of statistical adjustment for baselineimbalances in prognostic factors was investigatedin two primary studies, and in four additionalcomparisons (coronary angioplasty versus bypassgrafting, calcium antagonists for cardiovasculardisease, malaria vaccines and stroke unit care:comparisons 25–28 in Table 3). In two of the sixcomparisons there was evidence that adjustmentfor prognostic factors led to improvedconcordance of results between randomised andnon-randomised studies.MacLehose, Reeves, Harvey, Sheldon, Russelland Black 26MacLehose and colleagues restricted their reviewto studies where results of randomised and nonrandomisedcomparisons were reported togetherin a single paper, arguing that such comparisonsare more likely to be of ‘like-with-like’ than thosemade between studies reported in separate papers.They included primary studies and also reviewsthat pooled results from several individual studies.Of the 14 comparisons included in their report,three were based on reviews (comparisons 3, 7 and25 in Table 3) and the rest were results fromcomparisons within single studies. The nonrandomiseddesigns included comprehensivecohort studies, other observational studies andquasi-experimental designs.The ‘fairness’ or ‘quality’ of each of the comparisonsmade was assessed for comparability of patients,interventions and outcomes and additional studymethodology (see Table 4). Although the authorsdid not categorise comparisons as showingequivalence or discrepancy, the differences inresults were found to be significantly greater incomparisons ranked as being low quality.Benson and Hartz 32Benson and Hartz evaluated 19 treatmentcomparisons (eight in common with Britton andcolleagues 25 ) for which they located at least onerandomised and one observational study (definedas being a study where the treatment was notallocated for the purpose of research) in a search ofMEDLINE and the databases in the CochraneLibrary (Table 4). They only considered treatmentsadministered by physicians. Across the 19comparisons they found 53 observational and 83randomised studies, the results of which were metaanalysedseparately for each treatment comparison.Comparisons were made between the pooledestimates, noting whether the point estimate fromthe combined observational studies was within theconfidence interval of the RCTs. They found onlytwo instances where the observational andrandomised studies did not meet this criterion.Concato, Shah and Horwitz 33Concato and colleagues searched for meta-analysesof RCTs and of observational studies (restricted tocase–control and concurrent cohort studies)published in five leading general medical journals.They found only five comparisons where bothtypes of study had been meta-analysed [BCGvaccination for tuberculosis (TB), mammographicscreening for breast cancer mortality, cholesterollevels and death from trauma, treatment ofhypertension and stroke, treatment of hypertensionand coronary heart disease (CHD) (Table 3)]combining a total of 55 randomised and 44observational studies. They tabulated the results ofmeta-analyses of the randomised and theobservational studies and considered the similarityof the point estimates and the range of findingsfrom the individual studies. In all five instancesthey noted the pooled results of randomised andnon-randomised studies to be similar. Whereindividual study results were available, the range ofthe RCT results was greater than the range of theobservational results.Ioannidis, Haidich, Pappa, Pantazis, Kokori,Tektonidou, Contopoulous-Ioannidis and Lau 34Ioannidis and colleagues searched for reviews thatconsidered results of RCTs and non-randomisedstudies. In addition to searching MEDLINE theyincluded systematic reviews published in theCochrane Library, locating in total 45comparisons. Comparisons of RCTs with bothquasi-randomised and observational studies wereincluded. All meta-analytical results wereexpressed as odds ratios, and differences betweenrandomised and non-randomised resultsexpressed as a ratio of odds ratios and theirstatistical significance calculated. Findings acrossthe 45 topic areas were pooled incorporatingresults from 240 RCTs and 168 non-randomisedstudies. Larger treatment effects were noted moreoften in non-randomised studies. In 15 cases(33%) there was at least a twofold variation in oddsratios, whereas in 16% there were statisticallysignificant differences between the results ofrandomised and non-randomised studies. Theauthors also tested the heterogeneity of the resultsof the randomised and non-randomised studiesfor each topic. Significant heterogeneity was notedfor 23% of the reviews of RCTs and for 41% of thereviews of non-randomised studies.17© Queen’s Printer and Controller of HMSO 2003. All rights reserved.