Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27partially for the effect of the covariate. The logicalcorollary of this assumption is that it is alwaysworthwhile to adjust for a covariate, howeverpoorly it has been measured. However, this notionin turn depends critically on the assumption thatmisclassification in the covariate is itselfunbiased. 157 Greenland and Robins 162 discussedscenarios for case-control studies where thisassumption is relaxed. They considereddifferential misclassification in disease, exposureand a single covariate, and showed that in somemisclassification scenarios it can actually bedetrimental to adjust for the covariate – theunadjusted estimate may be closer to theunderlying effect than the adjusted estimate. Inthese situations it is not a matter of adjustmentbeing inefficient but rather that adjustment leadsto totally incorrect conclusions. Such an error willbe magnified by the additional credence oftengiven to an adjusted analysis when reporting studyresults.Greenland and Robins’ scenarios hint at amechanism that might explain our troublingfourth key result. However, their scenarios do nottranslate directly to consideration of nonrandomisedintervention studies: they consideredmeasurement error in the exposure (treatment)and case–control status (outcome). They also onlyassessed the impact of measurement error in asingle covariate, whereas case-mix adjustmentinvolves many covariates, all of which could besubject to misclassification. We have extended thehypothetical example in Table 34 to includestratification for a second covariate (CV2). InTable 34(c) we have introduced misclassification inCV2 but only for those misclassified for CV1. Weobserve that the OR adjusted for observed CV1and CV2 (OR = 1.32) is now further from theunderlying true value (OR = 0.5) than theunadjusted estimate (OR = 1.28). Hence such amechanism could explain our fourth observation,that adjustment in some situations can increasebias in estimates of treatment effects.If misclassification was solely a matter ofmeasurement error, such correlated misclassificationswould be rare. However, misclassification includesall processes that lead us to observe a value thatdiffers from the true value of the confounder. Forexample, consider a study in which the trueconfounder is a person’s average blood pressure.The blood pressure reading used in the analysiscould be misclassified owing to measurementerror, but it could also be misclassified owing tonatural within-person variation in blood pressurerelated to the time of day when the measurementwas made, or to a short period of stress. If asecond variable was included in the analysis thatwas also influenced by circadian rhythm or stress,misclassification in the second variable will becorrelated with misclassification in blood pressure.If case-mix adjustment models are beinginfluenced by correlated misclassification, it couldbe hypothesised that including more variables inan adjustment model will increase the chances ofcorrelated misclassifications, and hence increaseexpected bias. Comparison of the three differentlogistic models across Tables 24, 25, 27–29 and32 potentially supports this hypothesis: in nearlyall cases the logistic regression model includingthe most covariates (the ‘full model’) was morebiased than the logistic regression model with thefewest covariates (backward stepwise withp-to-remove of 0.05).Misclassification and measurement error incovariates has been cited as the possible root ofmany controversial associations detected throughobservational studies. 163 We are correspondinglyconcerned that measurement and misclassificationerrors may be largely responsible for our fourthresult.Differences between unconditional andconditional estimatesCovariates which are prognostic but balancedacross groups are rarely routinely adjusted for inanalyses of randomised controlled trials. Evenwhen adjustments are made, the adjustedestimates are often ignored when results of trialsare included in meta-analyses. We have followedthis standard approach in calculating the resultsof the RCTs that we created from the IST andECST data sets. These unadjusted estimates areknown as unconditional or population averageresults.Estimates of treatment effects where prognosticvariables have been adjusted for are known asconditional estimates, being conditional onknowledge of the covariates included in the analysis.Our comparison of results of RCTs with adjustedresults of non-randomised studies is thereforecomparing unconditional estimates of treatmenteffects from RCTs with conditional estimates oftreatment effects from non-randomised studies.When using ORs, it has been noted thatadjustment for prognostic covariates leads todifferences between unconditional and conditionalestimates of treatment effects, even for covariatesthat are balanced across treatment groups. 164,16583© Queen’s Printer and Controller of HMSO 2003. All rights reserved.