Evaluating non-randomised intervention studies - NIHR Health ...

Empirical estimates of bias associated with non-random allocation50through the recruitment of the trial. Figure 7illustrates this structure for the region-basedanalysis of the IST trial. Participants are classifiedaccording to the treatment they received (T or C),whether they were recruited in the first (subscriptB for before) or second (subscript A for after) halfof the trial, and according to the region number(subscripts 1–14).Historically controlled studies were constructedwithin each region by sampling participants whowere recruited to the control group during thefirst half of the trial and participants who wererecruited to the treatment group during thesecond half of the trial. For example, in NorthItaly, control participants were selected from cellC B1 and treated participants from cell T A1 .Concurrently controlled studies were constructedby selecting a region from which treatedparticipants were drawn and choosing controlparticipants from another region chosen atrandom. Participants were sampled regardless ofthe recruitment period. For example, aconcurrently controlled comparison could begenerated between treated patients in Scotland(sampling from cells T B4 and T A4 ) and controlpatients in Spain (sampling from C B12 and C A12 ). Itwas expected that no average bias would beobserved in large samples of concurrentlycontrolled studies generated in this manner asthere would be corresponding studies comparingtreated patients from Spain with control patientsfrom Scotland.The IST recruited sufficient participants forsamples of 100 treated and 100 controls to besampled from each region for each design. For theECST only samples of 40 treated and 40 controlswere drawn as sample sizes in regions were lower.In all instances random samples were drawn usingreplacement sampling techniques. 134 This partialbootstrap process ensures that the variabilitybetween samples is as close as possible to thatwhich would have been obtained if sampling froman infinite (or very large) population.For the purpose of comparing distributions ofresults, randomised trials of the same size as thenon-randomised studies were constructed usingthe same resampling methodology. RCTs forcomparison with historically controlled studieswere constructed by comparing treated patientssampled from the second half of the trial (thesame treated sample as selected for the HCT) withcontrol patients from the same region alsosampled from the second half of the trial. Forexample, a North Italy RCT for comparison with aNorth Italy HCT was constructed by samplingtreated participants from T A1 and controlparticipants from C A1 . RCTs for comparison withconcurrently controlled studies were constructedby comparing treated and control patientssampled from the same region regardless of therecruitment period. For example, a North ItalyRCT for comparison with concurrently controlledstudies was constructed by sampling treatedparticipants from T B1 and T A1 and controlparticipants from C B1 and C A1 (Figure 7).Historically controlled studies were constructed ineach of the 14 regions of the IST analysis and theeight regions of the ECST analysis. The wholeresampling process was repeated 1000 times inboth studies, generating results of 14,000 IST and8000 ECST historically controlled studies, and thesame number of comparable RCTs.Concurrently controlled studies wereconstructed for each of the 14 regions of the ISTanalysis and the eight regions of the ECSTanalysis. In addition, in an attempt to limit somesources of variability, concurrently controlledstudies were constructed using IST data from 10cities in the UK (see Appendix 8). All resamplinganalyses were repeated 1000 times, generating14,000 IST and 8000 ECST concurrentlycontrolled region-based comparisons and 10,000IST concurrently controlled UK city-basedcomparisons, all with the same number ofcomparable randomised trials.Statistical analysesThe focus of the statistical analysis was ondescribing and comparing the location (average)and spread (variability) of the treatment effectsobserved in studies of different designs. Treatmenteffects were calculated using standard methodscommonly reported in medical journals, andinterpreted according to their statisticalsignificance assessed at the 5% level usinga two-sided test.Observed outcomes of the resampled trialparticipants for each non-randomised andrandomised study were tabulated, and thetreatment effect expressed as an odds ratio (OR)with 95% confidence interval. For both IST andECST, OR < 1 indicate that the experimentaltreatment (IST, aspirin; ECST, carotidendarterectomy) had better outcomes thancontrols. The effect was deemed to be statisticallysignificant if the confidence interval excluded thenull hypothesis of no treatment effect (OR = 1).