Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27may lead to increases in average event rates.Historically controlled studies undertaken in sucha situation may be prone to underestimating thebenefits of treatment and may even falselyconclude that treatment does more harm thangood.The lack of systematic bias with the use ofgeographical controls is based on the presumptionthat geographical differences act in a haphazardmanner, and are as likely to lead to overestimatesof treatment effects as to underestimates. Therandom manner in which concurrent controlgroups were selected in the resampling exerciseensured that across a large number of studiesthese differences would be seen to balance eachother out, albeit possibly increasingunpredictability. This result does not indicate thatgeographically controlled studies are unbiased. Inreality, a single comparison between two areas islikely to be biased, as are meta-analyses ofseveral studies, although the direction in whichthe bias acts may be unknown. In addition,if an investigator chose a geographical controlgroup with knowledge of the likely differencesin case-mix, it would be possible for theselection to be manipulated (consciously orsubconsciously) in such a way as to introducea particular bias, akin to the bias observed inRCTs when treatment allocation is notconcealed. 20Similarly, we should consider whether themechanisms leading to unpredictability in bias,especially in studies generated from the IST, arelikely to apply widely across different clinical areas.Tables 38 and 40 in Appendix 8 show that the casemixof patients recruited to the IST variedbetween locations, both internationally andbetween cities in the UK. These haphazarddifferences, together with differences in otherunknown risk factors and aspects of patientmanagement and outcome assessment, will havecaused the unpredictability in the bias that wasobserved. Evidence is available in all areas ofmedicine that such differences exist, and thereforeit seems reasonable to conclude that theunpredictable behaviour in biases will be observedelsewhere, although the degree of unpredictabilitymay vary.Limitations of the resampling methodologyThe resampling method used participantsrecruited to a randomised controlled trial togenerate non-randomised studies. This, of course,is not what happens in reality, but there arereasons to believe that our approach is more likelyto have led to underestimates than overestimatesof bias.The degree of bias in a non-randomised studydepends on the similarity of the two groups fromwhich treated participants and controls are drawn.Sampling these groups from the same randomisedtrial is likely to have reduced such differences forthe following reasons:1. All participants included in the RCT will havebeen judged to have been suitable for eithertreatment. In a non-randomised studyparticipants who are suitable for only one ofthe two treatments may have been recruited tothat arm: there is usually no formal assessmentthat they would have been considered suitablefor the alternative. This difference will nearlyalways act to increase differences in outcomebetween the groups.2. The RCT was conducted according to aprotocol, describing methods for recruiting,assessing, treating and evaluating the patients.This will have reduced the variability withinthe trial. Although some non-randomisedstudies are organised using a protocol, manyare not.3. All participants included in the trial wererecruited prospectively. In non-randomisedstudies, especially those using historicalcontrols, participants are likely to have been‘retrospectively’ included in the study,potentially introducing additional bias.On balance, it could be argued that usingrandomly chosen international comparisons forselection of concurrent controls may be regardedas rather artificial and likely to have increaseddifferences between groups. In reality, aconcurrent control group in a non-randomisedstudy would be selected to minimise likelydifferences between groups, and such longdistancegeographical comparisons wouldprobably be avoided. It is perhaps more realistic tofocus on the magnitude of the biases observed inthe concurrent comparisons generated from theUK cities in the IST as being more representativeof what might occur in reality. The unsystematicbias seen here was less than that observed ininternational comparisons, but still large enoughto lead many studies falsely to obtain significantfindings of both benefit and harm.Importantly, we have concentrated on only oneaspect of quality in non-randomised studies: thereare other biases to which they are susceptible inthe same way as are RCTs.61© Queen’s Printer and Controller of HMSO 2003. All rights reserved.