Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27stratified estimate of the OR is >1 and inTable 33(c) the stratified estimate of the OR is >1.Misspecification of relationships for continuous orordinal covariates could therefore also (partly)explain the lack of adjustment we observed (Result2). It is also possible that misspecification of aU-shaped relationship as a dichotomy or a lineartrend could explain the increase in bias observedwith adjustment (Result 4). However, it seemsunlikely that this is the full explanation of theseresults as there are few true continuous or ordinalcovariates in the analyses, and U-shapedrelationships are rare.The second type of misspecification is theomission of interactions between covariates, orbetween covariates and allocation. If covariates dointeract, omitting the interaction term from acase-mix adjustment model will lead to areduction in the potential adjustment that can bemade, hence increasing residual confounding isanother possible explanation of our second result.Interactions between covariates and the treatmentallocation are more complicated, and imply thatthe treatment effect should not be summarised asa single value but as a set of values dependent onthe covariate. We have not considered this level ofcomplexity for either the results of RCTs or nonrandomisedstudies (despite this being likely forthe ECST, where degree of stenosis is known torelate to benefit of carotid endarterectomy). Whenthese interactions are omitted, the effect oftreatment is summarised as a single ‘average’value, the average depending on the distributionof covariates of participants included in the study.As the omission of interactions applied to both theRCTs and the non-randomised studies, it isunlikely to be the key explanation of the resultsthat we observed.Multicollinearity among covariatesIf a model includes two or more covariates whichare strongly correlated with each other, it isdifficult to disentangle their effects. In thissituation, it has often been noted that theestimated regression coefficients may changedrastically according to what other variables areincluded in the model, which is disconcerting. Inaddition, the estimates of effect are likely to bemade with low precision. Such multicollinearityamong covariates may not be too much of aproblem in a non-randomised study unless theycause the adjustment procedures to ‘explode’,producing infeasibly small or large estimates ofparameters or standard errors. However,multicollinearity between a covariate and thetreatment allocation will make it impossible toseparate the independent effects of the covariateand of treatment on outcome. 156 The allocationrelationships in non-randomised studies are rarelystrong enough for this to be a major concern, andthis is unlikely to be an explanation for the resultsthat we observed.Misclassification and measurement errorWe have already noted that in order to be able toadjust for a confounding factor, it must bemeasured and included in the adjustment model.To do this it is important (a) that the realconfounding factor is used, and not a surrogate orproxy variable, and (b) that it is measured withouterror. Covariate misclassification through the useof poor proxies, measurement error and withinparticipant instability in covariates (e.g. because ofcircadian rhythms) all lead to underestimation ofthe effect of each covariate on outcome. Thisphenomenon, often known as regression dilutionor attenuation, 157,158 is demonstrated usinghypothetical data in Table 34. The first rowpresents data from a non-randomised studystratified according to the true values of twocovariates, where the underlying treatment effectis OR = 0.5. Owing to the confounding effect ofthe first covariate, the observed result is biased:OR = 1.28 (Table 34a). Adjusting for the firstcovariate corrects for this bias (OR = 0.50).However, if the first covariate is assessed withmeasurement error, we could observe thecategorisation observed in Table 34(b). Here,adjustment for the same covariate has much lesseffect, the adjusted estimate being OR = 1.17.The impact of misclassification in this example islarge, but is generated by a typical degree ofmisclassification corresponding to a kappacoefficient of 0.4, which is routinely interpreted asshowing ‘fair to moderate’ agreement.Methods have been developed to correct forregression dilution and are widely used throughoutepidemiology. 158–160 However, they require thedegree of misclassification to be estimated, eitherwithin the study or in additional reliabilityinvestigations, and their ability to correct formisclassification depends on the precision withwhich these estimates are made. When there aremultiple sources of misclassification, correction formeasurement error becomes complex. 160,161Where confounders cannot be corrected formisclassification, it is often assumed thatadjustment using the covariate will adjust only81© Queen’s Printer and Controller of HMSO 2003. All rights reserved.