Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

Review of empirical comparisons of the results of randomised and non-randomised studies10and from contact with experts working in the field.The electronic search strategy used is provided inAppendix 1. Additionally, the searches carried outfor other sections of the project (includingsearches from the Cochrane Library databases)were screened to identify suitable papers forinclusion. Difficulties with devising searches formethodological topics are discussed in Chapter 4.The content, results and conclusions from each ofthe identified reviews were noted. In addition, themethodology of each review was critically assessedfor potential weaknesses. Aspects considered wereas follows:1. Was the identification of included studiesunlikely to be biased?2. Did the RCTs and non-randomised studiesrecruit similar participants, use similarinterventions and measure similar outcomes?3. Were the RCTs and non-randomised studiesshown to use similar study methodology in allrespects other than the allocation mechanism?4. Were sensible, objective criteria used todetermine differences or equivalence of studyfindings?ResultsEight reviews were identified which fulfilled theinclusion criteria; seven considered medicalinterventions and one psychological interventions.Brief descriptions of the methods and findings ofeach review are given below, with summary detailsgiven in Table 2. There is substantial overlap in theinterventions (and hence studies) that wereincluded in the reviews of medical interventions(Table 3).Description of the identified reviewsSacks, Chalmers and Smith 27Sacks and colleagues compared the results ofRCTs with historically controlled trials (HCTs).The studies were identified in Chalmers’ personalcollection of RCTs, HCTs and uncontrolled studiesmaintained since 1955 by searches of IndexMedicus, Current Contents and references of reviewsand papers in areas of particular medical interest(full list not stated). Six interventions were includedfor which at least two RCTs and two HCTs wereidentified [cirrhosis with oesophageal varices,coronary artery surgery, anticoagulants for acutemyocardial infarction, 5-fluorouracil adjuvanttherapy for colon cancer, bacille Calmette–Guérinvaccine (BCG) adjuvant immunotherapy anddiethylstilbestrol for habitual abortion (Table 3)].Trial results were classified as positive if there waseither a statistically significant benefit or if theauthors concluded benefit in the absence ofstatistical analysis, otherwise as negative. For eachof the six interventions, a higher percentage ofHCTs compared with RCTs concluded benefit:across all six interventions 20% of RCTs showedbenefit compared with 79% of the HCTs.The authors also considered whether statisticaladjustment for confounding factors in the HCTsreduced the discrepancy between the randomisedand non-randomised results. They found that theadjusted studies showed nearly the same treatmenteffect as the unadjusted studies.Kunz and Oxman 28 and Kunz, Vist and Oxman 29Kunz and Oxman searched the literature forreviews that made empirical comparisons betweenthe results of randomised and non-randomisedstudies. They included the results of the sixcomparisons in Sacks and colleagues’ study above,and results from a further five publishedcomparisons [antiarrhthymic therapy for atrialfibrillation, allogenic leucocyte immunotherapyfor recurrent miscarriage, contrast media forsalpingography, hormonal therapy forcryptorchidism, and transcutaneous electrical nervestimulation (TENS) for postoperative pain (Table 3)].In some of the comparisons, RCTs were comparedwith truly observational studies and, in others theywere compared with quasi-experimental trials. Aseparate publication of anticoagulants for acutemyocardial infarction already included in Sacksand colleagues’ review was also reviewed, 30 as was acomparison of differences in control group eventrates between randomised and non-randomisedstudies for treatments for six cancers (which doesnot fit within our inclusion criteria). 31 The reviewwas updated in 2002 including a further 11comparisons, and published as a Cochranemethodology review. 29The results of each empirical evaluation weredescribed, but no overall quantitative synthesis wascarried out. The results showed differencesbetween RCTs and non-randomised studies in 15of the 23 comparisons, but with inconsistency inthe direction and magnitude of the difference. Itwas noted that non-randomised studiesoverestimated more often than they underestimatedtreatment effects.Britton, McKee, Black, McPherson, Sandersonand Bain 25Britton and colleagues searched for primarypublications that made comparisons between
© Queen’s Printer and Controller of HMSO 2003. All rights reserved.TABLE 2 Description of studies comparing results of randomised and non-randomised studiesNumber ofcomparisonsOther studydesignsincludedNumbers ofRCTs/otherstudiesSummary ofoverallquantitativeresultsSacks, 1982 27 Kunz, 1998, 28 Britton, 1998 25 MacLehose, Benson, 2000 32 Concato, 2000 33 Ioannidis, 2001 34 Lipsey, 1996, 352002 29 (figures 2000 26 2001 36in parenthesesrefer to originalpublication)6 23 (11) 18 14 (38 outcomes) 19 5 45 76Historicallycontrolled trials(HCTs)50/56 263 (122)/246 (152) 46/41 31/68 83/53 55/44 240/168 Not stated79% of HCTsfound the therapyto be better thanthe controlregimen,compared to 20%of RCTsQuasi-experiments,historicallycontrolled trials(HCTs), patientpreference trialsIn 15 (9) of 23 (11)comparisons effectswere larger in nonrandomisedstudies,4 (1) studies hadcomparable results,whilst 4 (2)reported smallereffectsQuasi-experiments,naturalexperiments, andprospectiveobservationalstudiesQuasi-experimentaland observationalstudiesSignificant In 14 of 35differences werefound in 11 of 18comparisons, butthere wereinconsistencies inthe direction of thedifferencescomparisons thediscrepancy in RRwas 50%.Discrepancies weresmaller in “fairer”comparisonsObservationalstudiesOnly in 2 of 19comparisons didthe point estimateof observationalstudies lie outsidethe confidenceinterval for theRCTsCase–control andcohort studiesFor the five clinicaltopics consideredthe average resultsof the observationalstudies wereremarkably similarto those of theRCTsQuasi-experiments,cohort and case–control studiesORs of RCTs andnon-RCTs are highlycorrelated(r = 0.75).Differences beyondchance were notedin 16% ofcomparisons: twofolddiscrepancies inORs occurred in33%Non-randomisedcomparativestudiesMean effect sizes(SD) of 0.46 (0.28)for RCTs and 0.41(0.36) for nonrandomisedstudies. Meandifference in effectsizes was 0.5, butvaried between–0.6 and +0.8 SDscontinuedHealth Technology Assessment 2003; Vol. 7: No. 2711
Page 1 and 2: Health Technology Assessment 2003;
Page 3 and 4: Evaluating non-randomisedinterventi
Page 7: Health Technology Assessment 2003;
Page 27 and 28: © Queen’s Printer and Controller
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 72 and 73:
Empirical estimates of bias associa
Page 74 and 75:
Empirical estimates of bias associa
Page 76 and 77:
Empirical evaluation of the ability
Page 78 and 79:
Page 80 and 81:
Page 82 and 83:
70TABLE 22 Comparison of concurrent
Page 84 and 85:
Page 86 and 87:
74TABLE 26 Comparison of methods of
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
80TABLE 33 Hypothetical example dem
Page 94 and 95:
82TABLE 34 Hypothetical example dem
Page 96 and 97:
Page 98 and 99:
Page 100 and 101:
Discussion and conclusions88histori
Page 102 and 103:
Discussion and conclusions90For exa
Page 104 and 105:
Discussion and conclusionsNon-rando
Page 107 and 108:
Health Technology Assessment 2003;
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121:
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?