Evaluating non-randomised intervention studies - NIHR Health ...

Discussion and conclusions88historical and concurrent cohort studies. Logisticregression in fact tended to increase the bias.Propensity score methods performed slightly betterthan other methods, but did not yield satisfactoryadjustments in most situations. Detailedinvestigation revealed that adequate adjustment forselection bias could only be made when selectiondepends on a single prognostic factor that ismeasured and included in the adjustment model.Although apparent under-adjustment could beexplained by omission of important confoundingfactors, the observation that adjustment could alsoincrease bias required different explanations. Ofpossible explanations identified, we considered themost likely to be the difference betweenunconditional and conditional estimates of ORsand the inclusion of confounders in the adjustmentmodels that have correlated mismeasurement ormisclassification errors, and the differences betweenconditional and unconditional estimates.Discussion“Scientific evidence is commonly and properlygreeted with objections, scepticism and doubt.Responsible scientists are responsibly sceptical. Welook for failures of observation, gaps in reasoning andalternative explanations. This scepticism is itselfscrutinised. Scepticism must itself be justified,defended. One needs ‘grounds for doubt’.”From Rosenbaum 145Non-randomised studies are widely usedthroughout healthcare to evaluate the intendedeffects of healthcare interventions. They have beenincluded in many systematic reviews and, onoccasion, used as the sole basis for healthcaredecisions and policy. While they are widelyperceived to have greater ‘external validity’ thanRCTs, 25 their internal validity is questionable,principally owing to problems of selection bias,although they often have weaknesses in other areas.These weaknesses lead many to be sceptical aboutthe validity of their results. In this project we haveattempted to evaluate the degree to which thisscepticism is justified, through reviews of existingevidence and through new empirical investigations.Two studies recently published in the New EnglandJournal of Medicine (NEJM) 32,33 challenged theexisting evidence base that has accumulated aboutthe validity of non-randomised studies forassessing the intended effects of healthcareinterventions. These two studies and, to a largeextent, the research of the five other groupsreviewed in Chapter 3, attempted to answer thesimple question, ‘are non-randomised studiesbiased?’. Between them, these reviews haveaccumulated many instances where the results ofrandomised and non-randomised studies of thesame intervention are on average the same, butalso many examples where they differ. Althoughnot the conclusion of all the authors, the oneconclusion that fits with all previous research andis supported by our new empirical investigations(Chapter 6) is that non-randomised studies aresometimes but not always biased. Havingestablished that bias is a likely possibility, it is ofmore importance to understand (a) what causesbias in non-randomised studies, (b) how often andhow badly biased are the results from nonrandomisedstudies, (c) whether the presence ofbias can in any way be predicted and (d) what theimplications are for those producing, using andreviewing non-randomised studies.What are the causes of bias in nonrandomisedstudies?The principal difference between randomised andnon-randomised studies can be characterised as adifference in their susceptibility to selection bias,that is, a bias which acts such that participantsselected to receive one intervention are in someway different from those selected to receive thealternative intervention. Concealed randomisationspecifically removes the possibility of selectionbias, the only differences between the outcome ofdifferent groups being attributable to chance or tothe intervention, all else being equal. In nonrandomisedstudies, allocation to groups dependson other factors, sometimes known, sometimesunknown. When these factors are also related tooutcome, bias will be introduced.There are other issues that commonly lead to biasin non-randomised studies. For example, numbersof exclusions in non-randomised studies arefrequently unclear, treatment and outcomeassessment are rarely conducted according tostandardised protocols and outcomes may not beassessed blind. None of these issues areinsurmountable in a prospective non-randomisedstudy, but while the threat of increased selectionbias can be reduced, it can never be removed.How often and how biased are nonrandomisedstudies?Our resampling studies have demonstrated thatthe biases associated with historically andconcurrently controlled non-randomised studiesare large enough to impact on the conclusions of asystematic review. Some 50% of the concurrentlycontrolled non-randomised studies sampled fromthe IST met standard criteria for statistical