Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 21 Sampling probabilities use to generate non-randomised designs mimicking ‘allocation by indication’(a) Sampling probabilities by number of neurological deficits, IST0 1 2 3 4 5 6Aspirin 1.0 0.9 0.8 0.6 0.4 0.2 0.1Avoid aspirin 0.1 0.2 0.4 0.6 0.8 0.9 1.0(b) Sampling probabilities by consciousness on presentation, ISTFully alertDrowsy or unconsciousAspirin 1.0 0.5Avoid aspirin 0.5 1.0(c) Sampling probabilities by outcome, ISTAdverse outcomeNo adverse outcomeExperimental 0.75 1.00Control 1.00 0.75which patients have been allocated interventionsbased on their characteristics. These additionalanalyses were carried out on the IST data only.Non-randomised studies mimicking ‘allocation byindication’ were generated by identifying strongprognostic covariates in the IST and biasingallocation according to values of these covariates.These studies were based on a simplistic form ofallocation by indication, in which the decision totreat depends on a single observed covariate. Twodifferent covariates were considered separately:the number of neurological deficits and the levelof consciousness at admission. The prognosticvalue of these covariates is summarised inAppendix 8.To generate bias, participants within each regionwere divided into groups according to their valuesof the covariate and the treatment that theyreceived. For example, Figure 14 shows thecategorisation of participants in the IST accordingto whether they receive aspirin (T) or avoid aspirin(C), the number of neurological deficits with whichthey present (0–6, first subscript) and the regionwithin which they were recruited (1–14, secondsubscript). Participants were resampled from thisstructure with sampling probabilities determinedaccording to their allocation and covariate value.For both covariates sampling probabilities werechosen such that the treated group was weightedtowards participants with good prognosis, whereasthe control group was weighted towardsparticipants with poor prognosis, leading to anoverestimate of the benefit of experimentaltreatment. The sampling probabilities used withthe two covariates are given in Table 21(a) and (b).The same sampling probabilities were used in allregions.In practice, allocation by indication is morecomplicated than the single covariate allocationdescribed above – it depends on some unknowncombination of multiple pieces of covariateinformation that are predictive of prognosis, andis likely to vary both between individual cliniciansand over time. We wished to include studies withsuch a complex bias in our evaluation toinvestigate how well case-mix adjustment methodsmight work in more typical database-type analyses.Research into medical decision making hascharacterised some decision processes usingregression techniques to provide modelsexplaining how covariate information is combinedto make treatment decisions. Although it would bepossible to generate biased allocations using sucha decision model, we did not use this approach asit is too simplistic and artificial to test properly theability of case-mix adjustment methods to adjustfor selection bias. Such analyses wouldoverestimate the ability of case-mix adjustmentmethods to adjust for bias due to the circularity ofone regression model (the case-mix adjustmentmethod) being used to estimate parameters of anunderlying regression model (the allocationprocess). The many treatment decisions that didnot fit with the underlying regression model wouldnot be accounted for.The second approach that we used is based on theobservation that when decisions are made inrelation to prognosis, those decisions naturally67© Queen’s Printer and Controller of HMSO 2003. All rights reserved.