Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27Chapter 7Empirical evaluation of the ability of case-mixadjustment methodologies to control forselection biasIntroductionCase-mix (or risk) adjustment methods are usedwidely throughout healthcare research to enablecomparisons to be made between groups that arenot directly comparable owing to differences inprognostic (or confounding) factors. For example,comparisons between outcomes from differenthospitals are often confounded by the severity ofthe patients that they treat, and it is recommendedthat outcomes should be compared only whendifferences in case-mix (severity) are adjustedfor. 136,137 The philosophy of case-mix adjustmentis to ‘level the playing field’ during analysis,enabling a comparison of like-with-like to be madeat the point of analysis, even if comparable groupscould not be generated by the study design. Casemixadjustment is an attempt to achieve byanalysis what could not be done (or was not done)by the design.Consideration of case-mix is routinelyrecommended in the analysis of non-randomisedstudies. Guides for assessing the validity of nonrandomisedstudies for both therapy and harmrecommend readers to assess first whether‘investigators demonstrate similarity in all knowndeterminants of outcome’ and, if not, whetherthey ‘adjust for differences in analysis’. 138,139 Manyepidemiological and biostatistical texts advocateadjustment for differences in baseline covariateswhen they are observed, both in non-randomisedcontrolled studies and even in RCTs. 140–142We will focus on comparisons between two groups,which we refer to as treatment (or experimental)and control. In principle, many of the methodsdiscussed extend to multiple armed trials. Fourapproaches to dealing with differences in case-mixin non-randomised studies are commonlyencountered in the medical literature:1. Comparison of baseline characteristics.The baseline clinical and demographiccharacteristics of the two groups are comparedto ascertain whether differences in case-mixexist and hence determine the certainty withwhich the observed difference can be attributedto the intervention and not to confoundingfactors. This method does not attempt to adjustfor differences in case-mix but simply todiscover whether there is evidence ofconfounding and make inferences accordingly.The presence of baseline differences is usuallydetermined by tests of statistical significance,although such tests do not relate directly tocomparability. 1432. Standardised or stratified analyses. Studyparticipants are divided into groups (strata)with the same characteristics. Stratified analyseswork by making ‘like-with-like’ comparisonswithin each of these groups. The overalltreatment effect is calculated by computing aweighted average of the within-strata estimatesof the treatment effect. The most popularmethod is that of Mantel and Haenszel. 144Stratification is best used when there are onlyone or two baseline characteristics, and isfrequently used in epidemiological research tostandardise for differences in age and/or sex.3. Multiple regression. Regression models (linearregression if the outcome is continuous,logistic regression if it is binary and Coxmodels if censoring occurs) estimate how eachprognostic factor relates to outcome. When thecomparison between the groups is made,adjustments are added to or subtracted fromthe estimated treatment effect to account forthe impact of differences in each of thebaseline covariates according to their estimatedrelationship with outcome. The two stages ofthe process happen simultaneously, so that theresults depend on the correlations betweenbaseline characteristics and treatmentallocation. Stepwise regression procedures arecommonly used to identify covariates that aresignificantly related to outcome, and work in asequential manner such that adjustments areonly made for the subset of covariates thoughtto matter.4. Propensity score methods. Propensity scoresare the least familiar method. Whereas63© Queen’s Printer and Controller of HMSO 2003. All rights reserved.