Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 1 Sources of biasSource of bias RCTs Cohort studiesSelection bias Randomisation Control for confoundersPerformance bias Blinding (of participants and/or investigators) Measurement of exposureAttrition bias Completeness of follow-up Completeness of follow-upDetection bias Blinded outcome assessment Blinded outcome assessmentstudies may also be based on factors such asavailability of care or geographical location. Inobservational studies, therefore, there are likely tobe systematic differences in the case-mix ofpatients allocated to the intervention andcomparison groups.Allocation to groups can also be based on patientchoice, as in patient preference trials. 8 Individualpreferences for one treatment above another maywell imply differences in other consistent ways(potentially relating to prognosis), from those whodo not hold such a preference. 9 In addition,preference for a particular treatment may enhanceits therapeutic effect.Sources of bias in nonrandomisedstudiesThe Cochrane Collaboration handbook has laidout the four main sources of systematic bias intrials of the effects of healthcare as being selectionbias, performance bias, attrition bias and detectionbias (Table 1). All of these biases can affect nonrandomisedstudies and are discussed in thisreport. However, it is selection bias that we discussin the greatest detail, and evaluate in newempirical studies, as it is the potential for selectionbias that most clearly differentiates randomisedfrom non-randomised studies.Selection biasThe greatest distinction between the results ofrandomised and non-randomised studies is, asdescribed above, the risk of selection bias, wheresystematic differences in comparison groups ariseat baseline. The term selection bias can bemisleading as it is used to describe both biasedselection of participants for inclusion in a study(which applies to both experimental andobservational studies) and biased allocation ofpatients to a given intervention (which occurswhere randomisation is not used). The first type ofselection bias is usually classified as an issue ofexternal validity and is not discussed further inthis report. Rather, we consider the second type,which is an issue of internal validity. It is sometimesreferred to as case-mix bias, or confounding.In non-randomised studies, selection bias will beintroduced when participants chosen for oneintervention have different characteristics fromthose allocated to the alternative intervention (ornot treated). For observational studies the choiceof a given intervention is largely at the discretionof the treating clinician (as would occur in normalclinical practice). The choice of an interventionunder these circumstances will be influenced notonly by a clinician’s own personal preference forone intervention over another but also by patientpreference, patient characteristics and clinicalhistory. Sometimes the reasons for the choice of atreatment will be obvious, but at other times aclinician’s treatment decision will be influenced bysubtle clues that are not easily identifiable. 3 Thismay result in treatment groups that areincomparable, often with one intervention group‘heavily weighted by the more severely ill’. 10According to Miettinen, 11 in clinical practice (andtherefore in observational studies):“Interventions are commonly prompted by anindication, a state or event that signifies the prospectof an untoward outcome. Thus, by the very rationalityof decisions to intervene, the treated tend to differfrom the untreated with respect to their outlooks forthe outcome criterion in efficacy assessment; theretends to be confounding by the indication – usually suchthat the treated tend to have less favourable outcomethan the untreated.”In other words, when faced with a patient whomay be eligible to receive a given intervention,the decision to treat will be influenced by somefactor that in turn is related to the treatmentoutcome. This introduces systematic bias leadingto either over- or underestimates of treatmenteffects, depending on the treatment decisionmechanism.Confounding by indication can take several guisesand the term has been used to describe a numberof situations. The original definition refers to thesituation where an extraneous condition is both a3© Queen’s Printer and Controller of HMSO 2003. All rights reserved.