Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27Chapter 6Empirical estimates of bias associated withnon-random allocationIntroductionIn Chapters 4 and 5, many aspects of study qualitythat may be related to bias in non-randomisedstudies were identified. Two quality items wereemphasised that are specific to non-randomisedstudies rather than RCTs: bias introduced by nonrandomallocation and the use of case-mixadjustment methods. In this chapter, we report anempirical investigation into the impact of nonrandomallocation. In Chapter 7 we report asimilar investigation into the benefits of case-mixadjustment.Eight reviews were described in Chapter 3 thatattempted to evaluate the importance ofrandomisation. Our appraisal, however, identifiedproblems with their methodology, especially withregard to control for confounders (metaconfounding)and their inability to identify biasesother than those acting in a systematic manner.For our evaluation, we have chosen a differentmethodology – resampling – to overcome theseissues. By selectively sampling treated and controlparticipants from a completed large randomisedtrial it is possible to make comparisons betweengroups of individuals who did and did not receivethe intervention, but where the treatmentallocation was based not on randomisation but ona non-random mechanism. Effectively we areanswering the question, ‘what results would thetrialists have obtained if they had not comparedtheir treated participants with random controls butwith controls selected using a non-randomprocess, all other aspects of study design beingkept the same?’.Two classical non-randomised designs can beconstructed from the data of a multicentre trial:first, a concurrently controlled cohort study inwhich treated patients from one centre arecompared with an untreated control group from adifferent centre recruited during the same timeperiod; and second, a historically controlled studywhere the untreated controls are selected from thesame centre but from a period before the treatedpatients were recruited. The results from suchstudies can be compared directly with the resultsof the RCT in a single centre. Repeating thecomparisons using replacement samplingmethodology with data from many centres cangenerate distributions of results for randomisedand non-randomised studies. These distributionscan be compared to describe the impact of bothsystematic and unpredictable components of biasassociated with the allocation mechanism. Thecomparisons between study designs are based onthe same patient data, and hence across repeatedsamples will be unconfounded.We report the results of such resampling exercisesfor two large multicentre trials: the InternationalStroke Trial (IST) 132 and the European CarotidSurgery Trial (ECST). 133 The IST comparedmedical treatments in stroke patients to preventlong-term death or disability. The ECST was atrial of a surgical treatment in patients at risk ofstroke to prevent stroke or death. Full descriptionsare given in Appendix 8.Note that because of the adaptations made to thetrials for the purposes of the empiricalinvestigation (such as grouping of centres,omission of data from centres and periods, andselection of particular time-points for follow-up),the results reported here differ from those of theoriginal publications. 132,133 Notably, the full reportof the ECST trial reports treatment benefit inparticipants with high-grade stenosis, whereas thesimplified data set that we consider presents afinding of an average harmful effect. The originalpublications should be referred to for theappropriate analyses of these trials. 132,133MethodsGeneration of non-randomised andrandomised studiesThe resampling was undertaken in a similar wayfor both the IST and ECST. First, participantswere grouped into regions as described inAppendix 8. Secondly, participants were classifiedaccording to whether they were early or laterecruits to the trial, depending on whether theywere recruited before or after a time-point midway49© Queen’s Printer and Controller of HMSO 2003. All rights reserved.