Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

16TABLE 4 Details of methodology of empirical comparisons between randomised and non-randomised studies (cont’d)Method ofsummarisingstudyfindingsMethod forcomparingresultsbetweengroupsCriteria forcomparingvariability ofresultsSacks, 1982 27 Kunz, 1998, 28 Britton, 1998 25 MacLehose, Benson, 2000 32 Concato, 2000 33 Ioannidis, 2001 34 Lipsey, 1996, 352002 29 2000 26 2001 36Vote counting ofresults classified aspositive (eitherstatisticallysignificant or hadpositiveconclusions if nostatistical analysis)Comparison of thepercentage withpositive resultsNo consistentsummary: results ofrandomised andnon-randomisedgroups describedusing a variety ofmeasuresNo overall analysispresentedResults ofrandomised andnon-randomisedgroups describedusing riskdifferences, RRs orORsStatisticalsignificance of thedifference in effectsizesRRs and riskdifferencescalculatedseparately for MAsfor randomised andnon-randomisedstudiesDistribution ofrelative andabsolute differencesin results reportedCalculation ofoverall MA resultsand CIs (ORs forbinary data,differences inmeans forcontinuous)Assessment ofwhetherobservational pointestimate fell within95% CI for RCTsCalculation ofoverall MA resultsand CIs (RRs andORs)No overall analysispresentedNot assessed Not assessed Not assessed Not assessed Not assessed Dispersion of pointscalculated withoutconsideringdifferences insample sizeCalculation of fixedand random MAestimates expressedas ORs andlog ORsCalculation ofZ-scores fordifference betweentreatment effectsSignificance(p < 0.1) of tests ofbetween studyheterogeneity(1) Mean effectsizes of allrandomised and allnon-randomisedstudies.(2) distribution ofdifferences ineffect sizesbetweenrandomised andnon-randomised ineach MAGraph ofdistribution ofdifferences ofeffect sizesNot assessed,although standarddeviations ofrandomised andnon-randomisedstudies presented(unadjusted fordifferences insample size)Review of empirical comparisons of the results of randomised and non-randomised studiesCCTR, Cochrane Controlled Trials Register; MA, meta-analysis.
Health Technology Assessment 2003; Vol. 7: No. 27non-lymphatic leukaemia, the risk ratio in RCTswas 24 compared with 3.7 in non-randomisedstudies (comparison 23 in Table 3).The impact of statistical adjustment for baselineimbalances in prognostic factors was investigatedin two primary studies, and in four additionalcomparisons (coronary angioplasty versus bypassgrafting, calcium antagonists for cardiovasculardisease, malaria vaccines and stroke unit care:comparisons 25–28 in Table 3). In two of the sixcomparisons there was evidence that adjustmentfor prognostic factors led to improvedconcordance of results between randomised andnon-randomised studies.MacLehose, Reeves, Harvey, Sheldon, Russelland Black 26MacLehose and colleagues restricted their reviewto studies where results of randomised and nonrandomisedcomparisons were reported togetherin a single paper, arguing that such comparisonsare more likely to be of ‘like-with-like’ than thosemade between studies reported in separate papers.They included primary studies and also reviewsthat pooled results from several individual studies.Of the 14 comparisons included in their report,three were based on reviews (comparisons 3, 7 and25 in Table 3) and the rest were results fromcomparisons within single studies. The nonrandomiseddesigns included comprehensivecohort studies, other observational studies andquasi-experimental designs.The ‘fairness’ or ‘quality’ of each of the comparisonsmade was assessed for comparability of patients,interventions and outcomes and additional studymethodology (see Table 4). Although the authorsdid not categorise comparisons as showingequivalence or discrepancy, the differences inresults were found to be significantly greater incomparisons ranked as being low quality.Benson and Hartz 32Benson and Hartz evaluated 19 treatmentcomparisons (eight in common with Britton andcolleagues 25 ) for which they located at least onerandomised and one observational study (definedas being a study where the treatment was notallocated for the purpose of research) in a search ofMEDLINE and the databases in the CochraneLibrary (Table 4). They only considered treatmentsadministered by physicians. Across the 19comparisons they found 53 observational and 83randomised studies, the results of which were metaanalysedseparately for each treatment comparison.Comparisons were made between the pooledestimates, noting whether the point estimate fromthe combined observational studies was within theconfidence interval of the RCTs. They found onlytwo instances where the observational andrandomised studies did not meet this criterion.Concato, Shah and Horwitz 33Concato and colleagues searched for meta-analysesof RCTs and of observational studies (restricted tocase–control and concurrent cohort studies)published in five leading general medical journals.They found only five comparisons where bothtypes of study had been meta-analysed [BCGvaccination for tuberculosis (TB), mammographicscreening for breast cancer mortality, cholesterollevels and death from trauma, treatment ofhypertension and stroke, treatment of hypertensionand coronary heart disease (CHD) (Table 3)]combining a total of 55 randomised and 44observational studies. They tabulated the results ofmeta-analyses of the randomised and theobservational studies and considered the similarityof the point estimates and the range of findingsfrom the individual studies. In all five instancesthey noted the pooled results of randomised andnon-randomised studies to be similar. Whereindividual study results were available, the range ofthe RCT results was greater than the range of theobservational results.Ioannidis, Haidich, Pappa, Pantazis, Kokori,Tektonidou, Contopoulous-Ioannidis and Lau 34Ioannidis and colleagues searched for reviews thatconsidered results of RCTs and non-randomisedstudies. In addition to searching MEDLINE theyincluded systematic reviews published in theCochrane Library, locating in total 45comparisons. Comparisons of RCTs with bothquasi-randomised and observational studies wereincluded. All meta-analytical results wereexpressed as odds ratios, and differences betweenrandomised and non-randomised resultsexpressed as a ratio of odds ratios and theirstatistical significance calculated. Findings acrossthe 45 topic areas were pooled incorporatingresults from 240 RCTs and 168 non-randomisedstudies. Larger treatment effects were noted moreoften in non-randomised studies. In 15 cases(33%) there was at least a twofold variation in oddsratios, whereas in 16% there were statisticallysignificant differences between the results ofrandomised and non-randomised studies. Theauthors also tested the heterogeneity of the resultsof the randomised and non-randomised studiesfor each topic. Significant heterogeneity was notedfor 23% of the reviews of RCTs and for 41% of thereviews of non-randomised studies.17© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2: Health Technology Assessment 2003;
Page 3 and 4: Evaluating non-randomisedinterventi
Page 7: Health Technology Assessment 2003;
Page 23 and 24: © Queen’s Printer and Controller
Page 27: © Queen’s Printer and Controller
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 76 and 77: Empirical evaluation of the ability
Page 78 and 79:
Empirical evaluation of the ability
Page 80 and 81:
Page 82 and 83:
70TABLE 22 Comparison of concurrent
Page 84 and 85:
Page 86 and 87:
74TABLE 26 Comparison of methods of
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
80TABLE 33 Hypothetical example dem
Page 94 and 95:
82TABLE 34 Hypothetical example dem
Page 96 and 97:
Page 98 and 99:
Page 100 and 101:
Discussion and conclusions88histori
Page 102 and 103:
Discussion and conclusions90For exa
Page 104 and 105:
Discussion and conclusionsNon-rando
Page 107 and 108:
Health Technology Assessment 2003;
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121:
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

Create successful ePaper yourself

Delete template?

Save as template?