Evaluating non-randomised intervention studies - NIHR Health ...

Discussion and conclusions90For example, the biases observed in our empiricalinvestigations based on cohort designs could notbe removed using case-mix adjustment methods,even though we could describe the process bywhich allocation occurred (Chapter 7). This wasdespite having at our disposal complete data onseveral highly prognostic variables. Our principalfailure resulted from being unable to describe howallocation was linked to prognostic factors in amanner that could be adjusted for in a statisticalanalysis. Moses 18 stated that there are threerequirements for successful case-mix adjustment:(1) knowledge of which variables must be takeninto account, (2) measuring those variables on eachparticipant and (3) using those measurementsappropriately to adjust the treatment comparison.He suggested that researchers are likely to fail at allthree. Although it is well known that failure toidentify all relevant variables will lead to underadjustment(residual confounding), our evaluationshave suggested that failing to measure thecorrect variables properly and failing to modeltheir effects appropriately can lead to spuriousconclusions.The use of case-mix adjustment analyses shouldtherefore not be regarded as a guarantee that astudy is unbiased, or that the observed differencesin case-mix have been adjusted for. Investigatorsusing case-mix adjustment methodologies,especially logistic regression models, should drawconclusions cautiously, and probably regardadjusted analyses as exploratory rather thanconfirmatory. Our observations require furtherassessment and confirmation from investigationsin additional data sets to explore further themechanisms that are causing these failures.Nonetheless, they raise serious concern about theroutine promotion and use of case-mix adjustmentmethods for the analysis of non-randomisedstudies. We do not propose that case-mixadjustment be abandoned, but recommend thatgreater scepticism be applied in the interpretationof results.What are the implications for thoseproducing, reviewing and using nonrandomisedstudies?An investigator planning to undertake a nonrandomisedstudy should first make certain thatan RCT cannot be undertaken. 18 The advantagesof concealed randomised allocation are so greatcompared with the inadequacies that we havequantified for non-randomised studies that itshould be with the greatest reluctance that aninvestigator concludes that randomisation is notpossible. The ability to eradicate bias at thedesign stage is crucial to establishing thevalidity of a study. In particular, investigatorsshould not assume that statistical methods canbe used reliably to compensate for biasesintroduced through suboptimal allocationmethods.A prospective non-randomised study should beundertaken according to a protocol that iscarefully followed to ensure consistent inclusioncriteria, that all relevant factors are measuredaccurately for each participant and thatparticipants are all monitored in a standardmanner and blinded to treatment if possible. Insome situations it may even be possible to matchprospectively treated and control patients onimportant prognostic factors. 171 Byar pointed outthat the one piece of information required forsuccessful adjustment is knowledge of the reasonswhy each patient received their particulartreatment. 16 This information is usually notrecorded in non-randomised studies, and how itshould be measured is a real challenge worthy ofresearch. 18 As a minimum, recording details ofreasons for allocations of particular interventionsshould allow the subset of patients to be identifiedwho are not considered suitable for bothtreatments, and should not be included inanalyses. In addition, all prognostic variablesshould be measured in such a way as to allowmeasurement error, misclassification and withinpersonvariability to be assessed. Some authorshave attempted to acquire reliable risk assessmentsby retrospective case-note review, 172 but obtainingthese data prospectively has many attractions.Retrospective studies cannot use consistentinclusion criteria, or ensure that data arecomplete and consistently recorded. Phillipsand Davey Smith have pointed out that it isprobably more worthwhile to put effort intoundertaking a small observational study to ahigh standard than in obtaining poor qualityand inadequate data on a large number ofparticipants. 173 However, despite all these efforts,the possibility of residual confounding will not beremoved.Several authors have suggested additionalstrategies that investigators might consideras a way of identifying hidden bias. Rosenbaumproposed the routine use of extra controlgroups and the inclusion of additionaloutcomes that are known not to be alteredby the treatment as further checks oncomparability. 145,152 Several authors haveemphasised the use of sensitivity analyses toquantify the strength of confounding in a missing