Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27which raise concerns about the performance ofcase-mix adjustment methods:1. Comparisons between non-randomised groupsthat appear comparable in terms of case-mixare often biased, sometimes more than for nonrandomisedgroups that do not appearcomparable.2. Case-mix adjustment methods rarelyadequately adjust for differences in case-mix.3. Logistic regression always increases variabilityin study results.4. All adjustment methods can on occasionincrease systematic bias.The first and second observations are notsurprising, and have been discussed before. 150 Thethird observation has been demonstratedtheoretically always to be the case, 151 although wesuspect that this result is not well known. Thefourth observation is contrary to most beliefsabout case-mix adjustment methods, and has onlybecome detectable through the unique resamplingdesign of our investigation.As Rothman described above, statistical riskadjustment methods are alluring, appearing toprovide a simple solution to many of theinadequacies of the design and execution of nonrandomisedstudies. This message has been widelydisseminated throughout the medical researchcommunity, leading to their routine use inepidemiology and health services research. 136However, the validity of a risk-adjustment modeldepends on fulfilling a demanding set ofassumptions. Below we consider the assumptionsthat may be most critical.Why adjustment methods might notwork?Omitted covariatesRisk adjustment models can only adjust fordifferences in variables that have been observedand measured. As seen with the ‘allocation byindication’ mechanisms based on neurologicaldeficits (IST) (Table 30), if all variables linked tothe allocation mechanism have been measuredand observed, then adequate adjustment may bemade. However, in most situations we do not knowthe variables upon which allocation is based.There may be important prognostic factors thatthe investigators do not know about or have notmeasured which are unbalanced between groupsand responsible for differences in outcome. Forexample, when allocation may be influenced bylevel of consciousness (IST) (Table 31) but theconsciousness variable was not included in themodel, inadequate adjustment for bias was made.If the missing covariates affecting allocation arecorrelated with the observed covariates, somedegree of adjustment is likely to be observed (aswas seen for degree of consciousness), but it isunlikely to be adequate unless the correlations arevery strong. Many texts refer to the unadjustedeffect as ‘residual confounding’ or ‘hidden bias’.Rosenbaum proposed a strategy for investigatingthe robustness of an observational finding ofhidden bias based on sensitivity analyses whichdetermine the size of covariate associationrequired to nullify an observed treatment effect. 152In some situations this approach could help todecide whether hidden bias could fully explain anobserved effect, but such an assessment retains adegree of subjective judgement.Our first and second results could be explained bythe common situation of treatment assignmentdepending on unmeasured covariates. However,missing covariates cannot explain the third andfourth results.Misspecified continuous covariates and omittedinteractionsTwo forms of misspecification can occur in casemixadjustment analyses. The first is when acontinuous variable is categorised, or a categoricalvariable is regrouped into a smaller number ofcategories. Cochran 153 presented analyticalinvestigations that showed that for a continuouscovariate related to outcome with a monotonictrend, dichotomisation would leave 36% of thevariability caused by the relationship unexplained(subject to some distributional assumptions).His results suggest that five categories areneeded to explain successfully at least 90% of thevariability. Brenner and Blettner 154 extendedthis work to consider the efficiencies of differentapproaches of modelling various monotonictrends using multiple categorisations and linearterms.However, Brenner 155 also showed that if thecovariate does not have a monotonic effect, correctcategorisation of the data can be crucial toobtaining a sensible result. Table 33 presentshypothetical data taken from Brenner’s paperwhere there is a U-shaped relationship betweenthe covariate and the outcome. Adjusting for thecovariate classified in three categories (Table 33a)makes an appropriate adjustment, the observedtreatment effect being OR = 1 in all categories.However, if the risk factor is dichotomiseddifferent results are obtained depending on wherethe dichotomisation is made: in Table 33(b) the79© Queen’s Printer and Controller of HMSO 2003. All rights reserved.