Evaluating non-randomised intervention studies - NIHR Health ...

Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection bias68relate to patient outcome. To generate a biasrelated to prognosis, therefore, we can takeadvantage of the fact that the outcomes in thesepatients are already known, thus skipping theprocess of selecting a prognostic model to be usedto represent treatment decisions. We havetherefore generated a prognostic bias mimicking‘allocation by indication’ by stratifying patients ineach region according to outcome and treatment,and differentially sampling patients with good andbad outcomes in the two treatment arms. Thesampling probabilities used are given in Table 21(c).Probabilities were again chosen to lead tooverestimation of treatment benefit, and the samesampling probabilities were used in all samples.Correcting for this selection process is likely to bemore testing than correcting for real allocation byindication as it is based on information notdirectly available to the treatment allocator, whowill only have available signs, symptoms, historyand results of investigations on which to basetreatment.The resampling methods were repeated 1000times as for the historically and concurrentlycontrolled studies (Chapter 6). Thus, for the IST14,000 non-randomised studies (1000 for eachregion) were generated with bias related toneurological deficit score, 14,000 with bias relatedto level of consciousness and 14,000 with biasrelated to outcome, all of sample size 200. Theresults of these studies were compared with thoseof 14,000 RCTs of the same sample size sampledfrom the same data. The case-mix adjustmentmethods were applied to each of these 42,000non-randomised studies.Case-mix adjustment methodsEight case-mix adjustment strategies wereinvestigated: matching baseline groups,stratification, three variants of regression modelsand three propensity score methods.Exclusion of non-matching groupsThe first analysis investigated the degree to whichthe absence of differences in case-mix was relatedto the absence of selection bias. The significanceof baseline differences for the 10 prognosticvariables for IST and eight prognostic variablesfor ECST that are listed in Table 20 was testedusing t-tests (continuous variables), chi-squaredtests (binary and unordered categorical variables),and chi-squared tests for trend (orderedcategorical variables) using the STATA commandsttest, tabulate and mhodds, respectively.Studies were then classified by the number ofcovariates with significant (p < 0.05) baselinedifferences. Although we do not imagine thatresearchers would desist from analysing orpublishing results of non-randomised studies withsignificant differences in one or more baselinecovariates, this analysis allows us to investigate theadvice given to readers to assess whether‘investigators demonstrate similarity in all knowndeterminants of outcome’. 138,139Stratification on a single factorThe Mantel–Haenszel stratified method 144 wasapplied to all non-randomised studies for bothtrials. Stratification was undertaken according tothe most strongly prognostic factor for each of thetwo trial data sets: the neurological deficit scorewas used for IST (seven categories) and the degreeof stenosis for ECST (four categories). The STATAmhodds command was used to obtain an overallOR together with 95% confidence interval(calculated using the standard equation proposedby Robins and colleagues 148 ).Multiple logistic regression modelsMultiple logistic regression models were fitted tothe data from each non-randomised study for bothtrials. Three variants of the method were applied.The first included all prognostic variables listed inTable 20, and is termed the ‘full model’. For boththe IST and ECST studies a variable was alsoincluded indicating treatment group, the value ofwhich yielded the estimate of log OR (and 95%confidence interval) of the adjusted treatmenteffect. Analysis was done using the STATA logitcommand.Two stepwise alternatives to the full model wereconsidered, both using the STATA sw logitcommand. Both used backward selectiontechniques removing non-significant variablesfrom the full model one at a time. The modelsdiffered in the significance level used to decidewhether to remove a variable. The first methodremoved variables with p-values >0.15. Thesecond method used a stricter p-to-remove valueof 0.05, requiring greater evidence of arelationship between a covariate and outcome forthe covariate to be included in the final model. Inboth situations the variable for treatment groupwas forced to remain in the model throughout thestepwise procedure, its value again yielding theestimate of log OR (and 95% confidence interval)for the adjusted treatment effect.The logistic regression models estimated only themain effects associated with the covariates in themodel (as is typical of logistic regression analysespublished in the medical literature). No attempt