Evaluating non-randomised intervention studies - NIHR Health ...

Review of empirical comparisons of the results of randomised and non-randomised studies10and from contact with experts working in the field.The electronic search strategy used is provided inAppendix 1. Additionally, the searches carried outfor other sections of the project (includingsearches from the Cochrane Library databases)were screened to identify suitable papers forinclusion. Difficulties with devising searches formethodological topics are discussed in Chapter 4.The content, results and conclusions from each ofthe identified reviews were noted. In addition, themethodology of each review was critically assessedfor potential weaknesses. Aspects considered wereas follows:1. Was the identification of included studiesunlikely to be biased?2. Did the RCTs and non-randomised studiesrecruit similar participants, use similarinterventions and measure similar outcomes?3. Were the RCTs and non-randomised studiesshown to use similar study methodology in allrespects other than the allocation mechanism?4. Were sensible, objective criteria used todetermine differences or equivalence of studyfindings?ResultsEight reviews were identified which fulfilled theinclusion criteria; seven considered medicalinterventions and one psychological interventions.Brief descriptions of the methods and findings ofeach review are given below, with summary detailsgiven in Table 2. There is substantial overlap in theinterventions (and hence studies) that wereincluded in the reviews of medical interventions(Table 3).Description of the identified reviewsSacks, Chalmers and Smith 27Sacks and colleagues compared the results ofRCTs with historically controlled trials (HCTs).The studies were identified in Chalmers’ personalcollection of RCTs, HCTs and uncontrolled studiesmaintained since 1955 by searches of IndexMedicus, Current Contents and references of reviewsand papers in areas of particular medical interest(full list not stated). Six interventions were includedfor which at least two RCTs and two HCTs wereidentified [cirrhosis with oesophageal varices,coronary artery surgery, anticoagulants for acutemyocardial infarction, 5-fluorouracil adjuvanttherapy for colon cancer, bacille Calmette–Guérinvaccine (BCG) adjuvant immunotherapy anddiethylstilbestrol for habitual abortion (Table 3)].Trial results were classified as positive if there waseither a statistically significant benefit or if theauthors concluded benefit in the absence ofstatistical analysis, otherwise as negative. For eachof the six interventions, a higher percentage ofHCTs compared with RCTs concluded benefit:across all six interventions 20% of RCTs showedbenefit compared with 79% of the HCTs.The authors also considered whether statisticaladjustment for confounding factors in the HCTsreduced the discrepancy between the randomisedand non-randomised results. They found that theadjusted studies showed nearly the same treatmenteffect as the unadjusted studies.Kunz and Oxman 28 and Kunz, Vist and Oxman 29Kunz and Oxman searched the literature forreviews that made empirical comparisons betweenthe results of randomised and non-randomisedstudies. They included the results of the sixcomparisons in Sacks and colleagues’ study above,and results from a further five publishedcomparisons [antiarrhthymic therapy for atrialfibrillation, allogenic leucocyte immunotherapyfor recurrent miscarriage, contrast media forsalpingography, hormonal therapy forcryptorchidism, and transcutaneous electrical nervestimulation (TENS) for postoperative pain (Table 3)].In some of the comparisons, RCTs were comparedwith truly observational studies and, in others theywere compared with quasi-experimental trials. Aseparate publication of anticoagulants for acutemyocardial infarction already included in Sacksand colleagues’ review was also reviewed, 30 as was acomparison of differences in control group eventrates between randomised and non-randomisedstudies for treatments for six cancers (which doesnot fit within our inclusion criteria). 31 The reviewwas updated in 2002 including a further 11comparisons, and published as a Cochranemethodology review. 29The results of each empirical evaluation weredescribed, but no overall quantitative synthesis wascarried out. The results showed differencesbetween RCTs and non-randomised studies in 15of the 23 comparisons, but with inconsistency inthe direction and magnitude of the difference. Itwas noted that non-randomised studiesoverestimated more often than they underestimatedtreatment effects.Britton, McKee, Black, McPherson, Sandersonand Bain 25Britton and colleagues searched for primarypublications that made comparisons between