Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

Empirical estimates of bias associated with non-random allocation5020105Odds ratio210.50.20.10.050.02RCTsCCsFIGURE 10 Comparison of distribution of results of 10,000 concurrently controlled studies and 10,000 RCTs resampled from 10 UKcities within the ISTcomparable RCTs. Again, the increase in variabilityof results for concurrently controlled studies wasevident, the distribution of results being 1.8 timeswider than for RCTs, slightly less than for theregional IST comparisons (Table 15). Some45% of concurrently controlled results werestatistically significant compared with 13% ofRCTs, the significant results being evenlydistributed between benefit (22%) and harm(23%).1002020105Odds ratio210.50.20.10.050.020.01RCTsCCs54FIGURE 11 Comparison of distribution of results of 8000 concurrently controlled studies and 8000 RCTs resampled from eight regionswithin the ECST
Health Technology Assessment 2003; Vol. 7: No. 27Results from the ECST region-based analysisFigure 11 displays results of 8000 concurrentlycontrolled comparisons and 8000 RCTs generatedfrom the ECST data set. The pattern of results ofconcurrent controlled studies are similar to thoseof RCTs. The concurrently controlled comparisonswere not more variable than the RCTs; theirdistribution was only 1.01 times as wide as theRCTs (Table 15); 9% of both concurrentlycontrolled studies and RCTs were statisticallysignificant.Historically controlled studiescompared with randomised controlledtrialsTable 16 shows results of comparisons ofhistorically controlled studies and RCTs sampledfrom the IST and ECST. The data from thehistorically controlled studies generated from thetwo trials display different patterns.Results from the IST region-based analysisIn the overall results of the historically controlledstudies from the IST there was no evidence ofsystematic bias. However, there was an increase inunpredictability in study results; the distribution ofhistorically controlled results was 20% wider thanthat for the RCTs (Table 16). This increase isdiscernible in Figure 12, where the results of the14,000 historically controlled studies and 14,000comparable RCTs are displayed. The increasedunpredictability increased the percentage ofstudies deemed statistically significant from 11 to20%, with increases in the number of findings ofboth statistically significant benefit and harm.Table 17 presents the same results broken downaccording to region. The results disaggregated atthis level appeared rather different. For eachregion there was little evidence of an increase inunpredictability (the SDs of the historicallycontrolled studies were all within 7% of the SD ofthe RCTs). However, there was evidence ofsystematic bias within many of the regions,although the magnitude and direction of the biasvary. For example, in Scotland the average OR forhistorically controlled studies was 1.23 comparedwith an average OR of RCTs of 0.78. In contrast toScotland, in Sweden the average OR fromhistorically controlled studies was 0.44 comparedwith 0.94 from the RCTs, 81% of historicallycontrolled studies concluding statisticallysignificant benefit. When aggregated across theregions, these varying systematic biases on averagecancel out and manifest as an increase inunpredictability.Results from the ECST region-based analysisThe overall results of the historical controlledstudies sampled from the ECST analyses (Table 16)showed a large systematic bias, but with littleincrease in unpredictability. Whereas the averageOR in the 8000 RCTs was 1.23, the average OR ofthe 8000 historically controlled studies was 1.06.This difference is noticeable in the comparison ofthe distribution of the results of the studies inFigure 13. The results of the two study designsalso show similar levels of variability. Thesystematic bias impacted on statistical significanceby shifting the distribution of results in onedirection: 6% of the historically controlled studiesconcluding significant benefit from carotidsurgery compared with 4% of the RCTs, whereas10% of the historically controlled studiesconcluded significant harm compared with12% of the RCTs.TABLE 16 Comparisons of results of RCTs with results of historically controlled non-randomised studies based on analyses of 1000 setsof resampled studiesPercentage of studies with statisticallysignificant results showing beneficial orharmful effects of the intervention (p < 0.05)Average OR SD of log OR Beneficial Harmful TotalIST – 14 regionsRCTs 0.89 0.35 9 2 11Historical controls 0.88 0.44 16 4 20Ratio of SDs1.2-foldECST – 8 regions aRCTs 1.23 0.83 4 12 16Historical controls 1.06 0.85 6 10 16Ratio of SDs1.03-folda All patients entering the trial after 1990 were excluded when the protocol’s inclusion criteria were changed.55© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2:
Health Technology Assessment 2003;
Page 3 and 4:
Evaluating non-randomisedinterventi
Page 5 and 6:
Page 7:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16: Health Technology Assessment 2003;
Page 17: Health Technology Assessment 2003;
Page 23 and 24: © Queen’s Printer and Controller
Page 27 and 28: © Queen’s Printer and Controller
Page 33: Health Technology Assessment 2003;
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 76 and 77: Empirical evaluation of the ability
Page 82 and 83: 70TABLE 22 Comparison of concurrent
Page 86 and 87: 74TABLE 26 Comparison of methods of
Page 92 and 93: 80TABLE 33 Hypothetical example dem
Page 94 and 95: 82TABLE 34 Hypothetical example dem
Page 100 and 101: Discussion and conclusions88histori
Page 102 and 103: Discussion and conclusions90For exa
Page 104 and 105: Discussion and conclusionsNon-rando
Page 117 and 118:
Page 119 and 120:
Page 121:
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?