Evaluating non-randomised intervention studies - NIHR Health ...

Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection bias64regression models described in (3) model howcovariates relate to outcome, propensity scoremethods model how the same covariates relateto treatment allocation. 145–147 The principle isthat the propensity score summarises themanner in which baseline characteristics areassociated with treatment allocation, so thatselection bias is removed when comparisonsare made between groups with similarpropensity scores. 146 The method involvescalculation for each individual in the datasetthe propensity probability that estimates theirchance of receiving the experimentalintervention from their baseline characteristics.In an RCT, where there should be norelationship between baseline characteristicsand treatment assignment, this probability willbe the same for each participant (e.g. 0.5 if thegroups are of equal size). In non-randomisedstudies, it is likely that treatment assignmentdoes depend on baseline covariates, and thatpropensity scores will vary betweenindividuals. For example, patients with moresevere disease may be more likely to receiveone treatment than the other. Propensityscores in such a situation would relate todisease severity, and the average propensityscore in the experimental group will differfrom the average in the control group.Estimation of propensity scores is typicallyundertaken using multiple logistic regressionmodels, the outcome variable being treatmentallocation.What evidence is there that these methods actuallyadjust for selection bias in non-randomisedstudies? Although there is plenty of literaturedemonstrating that case-mix adjustment canchange estimates of treatment effects, none of thetexts that we have consulted cite any empiricalevidence demonstrating that case-mix adjustment,on average, reduces bias. Broader searches of themethodological literature related to case-mixlikewise did not identify supporting empiricalevidence.To the contrary, there are hints in the literaturethat case-mix adjustment methods may notadequately perform the task to which they areapplied. Of the eight reviews in Chapter 3, twocompared adjusted and unadjusted estimates oftreatment effects from non-randomised studieswith the results of similar RCTs, and noted thatthere was little evidence that adjustmentconsistently moved the estimates of treatmenteffects from non-randomised studies towards thoseof RCTs. 25,27In an attempt to provide empirical evidence ofthe value of case-mix adjustment, we have usedthe same non-randomised studies generated byresampling participants from the IST 132 andECST 133 data sets (described in Chapter 6) toevaluate the performance of eight different casemixadjustment methods in controlling forselection bias. As explained previously,resampling of data from the trials was used togenerate randomised and non-randomisedstudies of different designs in such a way that thedifferences between their results with respect tolocation and spread could only be attributed toselection bias. Case-mix adjustment methodswere then applied to each resampled nonrandomisedstudy making use of the availablebaseline data (different in the two trials – seeTable 20). The distribution of results of theadjusted non-randomised studies was thencompared with both the distribution of unadjustedresults and the distribution of the results of theRCTs, to see whether there was any evidence thatadjustment had reduced or removed the selectionbias.In our investigations we evaluate the ability ofcase-mix adjustment to control for three differenttypes of bias:1. Naturally occurring systematic bias, whereresults are consistently either all overestimatesor all underestimates of the treatment owing tosome naturally occurring unknown allocationmechanism (as was observed in the ECSThistorical comparisons in Chapter 6).2. Naturally occurring unpredictable bias, whereresults are biased variably with differentmagnitudes in different directions leading toboth underestimates and overestimates oftreatment effects (as was observed in the ISTconcurrent comparisons in Chapter 6).3. Bias arising where allocation to treatment isprobabilistically linked to a prognostic variable,such that participants are more likely or lesslikely to receive treatment according to theirobserved characteristics. This in fact is a modelof practice in much of medicine, wheretreatment decisions are made according to theconditions and characteristics that each patientdisplays (allocation by indication). Although wehave no realistic way of mimicking such amechanism for the two trial data sets, wegenerate two artificial scenarios when allocationrelated (a) to the value of a single prognosticcovariate, and, more realistically, (b) to anunknown function of a set of prognosticcovariates.