Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 271002020105Odds ratio210.50.20.10.050.020.01RCTsHCTsFIGURE 13 Comparison of distribution of 8000 historically controlled studies and 8000 RCTs resampled from eight regions withinthe ECSTTABLE 18 Comparisons of results of RCTs with results of historically controlled non-randomised studies based 1000 sets of resampledstudies analysed by region (ECST)Percentage of studies withstatistically significantresults (p < 0.05):Average OR SD log OR benefit/harmRegion RCT HCT Ratio RCT HCT Ratio RCT HCTRegion 1 1.53 3.24 2.12 0.52 0.60 1.15 0/14 0/54Region 2 0.78 0.79 1.01 0.57 0.56 0.98 7/1 6/1Region 3 0.71 0.64 0.90 0.58 0.55 0.95 7/0 11/0Region 4 0.52 0.42 0.81 0.68 0.68 1.00 13/0 25/0Region 5 2.68 2.04 0.76 0.72 0.67 0.93 0/24 0/16Region 6 1.20 0.87 0.73 0.54 0.52 0.96 1/4 4/1Region 7 1.28 0.88 0.69 0.59 0.53 0.90 0/6 4/2Region 8 2.85 1.47 0.52 0.57 0.50 0.88 0/47 0/10Total 1.23 1.06 0.86 0.83 0.85 1.02 4/12 6/10these regions, but very similar heterogeneity wasseen between the results of the historicallycontrolled trials.DiscussionOur evaluations have detected the bias in nonrandomisedstudies using either concurrent orhistorical controls. Compared with the use ofrandomised controls, we have shown that use ofnon-randomised controls increases the probabilityof statistically significant study findings (and hencethe likely conclusions), in addition to altering theestimates of treatment effects. The bias observedin the non-randomised studies acted in two ways,which we refer to as systematic and unpredictablecomponents of bias.Systematic biasSystematic bias acts consistently in a givendirection leading to differences in average overall57© Queen’s Printer and Controller of HMSO 2003. All rights reserved.