Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

Review of empirical comparisons of the results of randomised and non-randomised studies20to be expected. It is also possible that the mostextreme RCT results arose in particularly smallrandomised studies, and that there is an absenceof equivalently small observational studies.Ioannidis and colleagues investigated thestatistical significance of the heterogeneity. 34 Thisanalysis is also dependent on the sample size andnumber of studies considered, hence thecomparison between randomised and nonrandomisedstudies may be confounded.DiscussionThe conclusions of the eight reviews are divergent,and as all the reviews have weaknesses, it isdifficult to draw conclusions concerning theimportance of randomisation from theseinvestigations. The only robust conclusion that canbe drawn is that in some circumstances the resultsof randomised and non-randomised studies differ,but it cannot be proved that differences are notdue to other confounding factors. The frequency,size and direction of the biases cannot be judgedreliably from the information presented.One review reported the existence of a systematicbias, namely that historically controlled trials weremore likely to conclude benefit of new therapiesthan RCTs. 27 However, as the conclusions werebased on statistical significance and not size ofeffect, this may be confounded by sample size.Direct comparison of particular non-randomiseddesigns with RCTs was not made in any of theadditional seven reviews.Two reviews raised concerns about the usefulnessof case-mix adjustment methods in nonrandomisedstudies. 25,27 Again, the validity ofthese conclusions is uncertain owing to thepossibility of meta-confounding.The efforts used to control confounding variedbetween the reviews. The potential forconfounding in these comparisons requiresdetailed, clinically informed assessment of thesimilarity of populations, interventions andoutcome assessments used in the RCTs and nonrandomisedstudies. Six of the eight reviews arejudged to be of poor quality in this regard, as theymade no or little assessment of comparability. Thereviews by Britton and colleagues 25 and MacLehoseand colleagues 26 used the most comprehensiveassessments to judge comparability, especially forthe six comparisons they discussed in detail. Fortwo of the four interventions considered in Brittonand colleagues’ report, the only conclusion thatcould be drawn was that due to differences in era,populations, dosage and length of follow-up,differences between randomised and nonrandomisedstudies were to be expected. Brittonand colleagues implied that for these clinicalinterventions a ‘fair’ comparison of randomisedand non-randomised studies would not bepossible. MacLehose and colleagues concludedtheir review by recommending that onlycomprehensive cohort studies could make faircomparisons between randomised and nonrandomisedevidence. Few such studies have everbeen undertaken.These investigations also raise concerns regardingthe usefulness of meta-epidemiologicalinvestigations where there is (or could be)variability in the direction of bias. The metaepidemiologicalmethod is designed to identifyand estimate only systematic bias. Only if theselection of groups in non-randomised studiesconsistently led to over-representation of high- (orlow-) risk participants in one group over anotherwould the bias always be seen to act in the samedirection. If selection bias in non-randomisedstudies arises as a result of haphazard variations incase-mix, there will be a mixture of under- andoverestimates of the treatment effect. The resultsmight all be biased, but not all in the samedirection. In these circumstances, the differencebetween randomised and non-randomised studieswould manifest as an increase in variance orheterogeneity of treatment effect (beyond thatexpected by chance) rather than (or in additionto) a systematic bias. This possibility is perhapshinted at by the observation in five of the eightreviews that the differences observed betweenrandomised and non-randomised studies did notact in a consistent manner. Ideally, a formalstatistical comparison should aim to compare theheterogeneity in treatment effects, and not just theaverage treatment effects between randomised andnon-randomised groups. None of the eightcomparisons used a statistical model to compareresults and, in any case, even the random effectsmethod presently recommended for metaepidemiologicalstudies would be inadequate as itassumes that the variance of effects in all groupsare equal. 22These major problems are in contrast to thesuccessful application of the meta-epidemiologicalmethod to understand the importance ofparticular design features for RCTs. 20,21,23,43 Thepotential for meta-confounding in RCTs may bereduced if the types of participants, interventionsand outcomes are more similar across trials than
Health Technology Assessment 2003; Vol. 7: No. 27across non-randomised studies, and there arefewer methodological characteristics of interest(randomisation, concealment of allocation, blindingand completeness of follow-up). Also, there areclear reasons to believe that the biases againstwhich these design features protect are likely to actin favour of a particular treatment (theexperimental treatment), leading to systematic bias,such that the problem of increased heterogeneitybetween study designs does not arise.To take account of our concerns about thecomplexity of biases in non-randomised studies,we have first undertaken a review of qualitydomains and assessment methods used to considerthe likelihood of bias in non-randomised studies(Chapters 4 and 5). Second, we have undertakennovel methodological assessments of theimportance of two quality issues which are specificto non-randomised studies: method of allocation(Chapter 6) and use of case-mix adjustment(Chapter 7). Our methodological assessments havebeen designed to overcome the particularproblems of meta-confounding and variability ofdirection and size of bias that plague theinterpretation of these previous reviews.21© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2: Health Technology Assessment 2003;
Page 3 and 4: Evaluating non-randomisedinterventi
Page 7: Health Technology Assessment 2003;
Page 23 and 24: © Queen’s Printer and Controller
Page 27 and 28: © Queen’s Printer and Controller
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 76 and 77: Empirical evaluation of the ability
Page 82 and 83:
70TABLE 22 Comparison of concurrent
Page 84 and 85:
Empirical evaluation of the ability
Page 86 and 87:
74TABLE 26 Comparison of methods of
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
80TABLE 33 Hypothetical example dem
Page 94 and 95:
82TABLE 34 Hypothetical example dem
Page 96 and 97:
Page 98 and 99:
Page 100 and 101:
Discussion and conclusions88histori
Page 102 and 103:
Discussion and conclusions90For exa
Page 104 and 105:
Discussion and conclusionsNon-rando
Page 107 and 108:
Health Technology Assessment 2003;
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121:
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?