Evaluating non-randomised intervention studies - NIHR Health ...

Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection biasdirectly the likely degree of residual confoundingthat may be present, and therefore we cannotgauge how biased adjusted results may still be. Bycomparing with results based on randomisation,our investigations suggest that the degree ofunderadjustment may be large. Indeed, ourresults may in fact be overoptimistic, as thecovariate data used were recorded in a standardway according to trial protocols, and werecomplete for all participants. In many nonrandomisedstudies measurement methods arenot standardised. Also, covariate data areincomplete (especially in retrospective studies),leading to bias if the observations are not missingat random.Our two greatest concerns are the potentialincrease in bias that could occur as a result of theexistence of correlated misclassification ofcovariates, and the differences betweenconditional and unconditional estimates.Correlated misclassification is a problem inherentto the data, and cannot be adjusted for. It is verydifficult to know the degree of misclassificationand error in a variable, and impossible to knowwhether the variable being used is the ‘true’confounder or just a proxy. These findingsquestion the appropriateness of the strategy ofincluding data on all available potentialconfounders when adjusting for case-mix, whichhas been the starting point of many riskadjustmentmethods used throughout healthcare.However, the same findings could be explained bythe peculiar differences between unconditionaland conditional estimates of treatment effectsobserved when results are expressed as ORs,although this mechanism only applied to estimatesobtained from logistic regression and stratificationmethods.The finding of high levels of residual confoundingand the detrimental effect of adjustment were seenin both historically controlled studies, known to beprone to systematic bias, and in concurrentlycontrolled studies, more prone to unpredictabilityin bias. The relationships were also noted instudies mimicking allocation by indication.It is important to find out whether suchdestructive relationships between covariates arecommon. We have examined data from only twoclinical situations, but in both we observed resultsthat undermine the use of case-mix adjustment.Also in the IST, case-mix adjustment was found tobe detrimental in eight of the 14 regions.There appears to be a small potential benefit ofusing propensity score methods over logisticregression for case-mix adjustment in terms of theconsistency of estimates of treatment effects. Whilelogistic regression always increased the range ofobserved treatment effects, propensity scoremethods did not. This finding may indicate agreater role for propensity score methods inhealthcare research, although in the particularapplications investigated neither approachperformed adequately.For those critically appraising non-randomisedstudies, the recommendation to assess whether“investigators demonstrate similarity in all knowndeterminants of outcome” 138,139 has not beenuniversally supported by our empiricalinvestigations. The second recommendation, toassess whether they “adjust for these differencesin analysis” is also not supported empirically.Our analyses suggest that there are considerablecomplexities in assessing whether a casemixadjustment analysis will increase ordecrease bias.These findings may have a major impact on thecertainty which we assign to many effects inhealthcare which have been made on the basis ofusing risk adjustment methods.86