Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27Results from the ECST region-based analysisFigure 11 displays results of 8000 concurrentlycontrolled comparisons and 8000 RCTs generatedfrom the ECST data set. The pattern of results ofconcurrent controlled studies are similar to thoseof RCTs. The concurrently controlled comparisonswere not more variable than the RCTs; theirdistribution was only 1.01 times as wide as theRCTs (Table 15); 9% of both concurrentlycontrolled studies and RCTs were statisticallysignificant.Historically controlled studiescompared with randomised controlledtrialsTable 16 shows results of comparisons ofhistorically controlled studies and RCTs sampledfrom the IST and ECST. The data from thehistorically controlled studies generated from thetwo trials display different patterns.Results from the IST region-based analysisIn the overall results of the historically controlledstudies from the IST there was no evidence ofsystematic bias. However, there was an increase inunpredictability in study results; the distribution ofhistorically controlled results was 20% wider thanthat for the RCTs (Table 16). This increase isdiscernible in Figure 12, where the results of the14,000 historically controlled studies and 14,000comparable RCTs are displayed. The increasedunpredictability increased the percentage ofstudies deemed statistically significant from 11 to20%, with increases in the number of findings ofboth statistically significant benefit and harm.Table 17 presents the same results broken downaccording to region. The results disaggregated atthis level appeared rather different. For eachregion there was little evidence of an increase inunpredictability (the SDs of the historicallycontrolled studies were all within 7% of the SD ofthe RCTs). However, there was evidence ofsystematic bias within many of the regions,although the magnitude and direction of the biasvary. For example, in Scotland the average OR forhistorically controlled studies was 1.23 comparedwith an average OR of RCTs of 0.78. In contrast toScotland, in Sweden the average OR fromhistorically controlled studies was 0.44 comparedwith 0.94 from the RCTs, 81% of historicallycontrolled studies concluding statisticallysignificant benefit. When aggregated across theregions, these varying systematic biases on averagecancel out and manifest as an increase inunpredictability.Results from the ECST region-based analysisThe overall results of the historical controlledstudies sampled from the ECST analyses (Table 16)showed a large systematic bias, but with littleincrease in unpredictability. Whereas the averageOR in the 8000 RCTs was 1.23, the average OR ofthe 8000 historically controlled studies was 1.06.This difference is noticeable in the comparison ofthe distribution of the results of the studies inFigure 13. The results of the two study designsalso show similar levels of variability. Thesystematic bias impacted on statistical significanceby shifting the distribution of results in onedirection: 6% of the historically controlled studiesconcluding significant benefit from carotidsurgery compared with 4% of the RCTs, whereas10% of the historically controlled studiesconcluded significant harm compared with12% of the RCTs.TABLE 16 Comparisons of results of RCTs with results of historically controlled non-randomised studies based on analyses of 1000 setsof resampled studiesPercentage of studies with statisticallysignificant results showing beneficial orharmful effects of the intervention (p < 0.05)Average OR SD of log OR Beneficial Harmful TotalIST – 14 regionsRCTs 0.89 0.35 9 2 11Historical controls 0.88 0.44 16 4 20Ratio of SDs1.2-foldECST – 8 regions aRCTs 1.23 0.83 4 12 16Historical controls 1.06 0.85 6 10 16Ratio of SDs1.03-folda All patients entering the trial after 1990 were excluded when the protocol’s inclusion criteria were changed.55© Queen’s Printer and Controller of HMSO 2003. All rights reserved.