Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

Introduction4clinical indication for the application of theintervention under study and an indication for theoutcome under study. For example, arrestedlabour is a factor that may directly contribute tothe decision to introduce fetal monitoring, that is,arrested labour is an indication for the monitoringprocedure. Any observational study would beexpected to show a higher proportion of arrestedlabours among the monitored participants. Sincearrested labour is also an indication for Caesareansection, we would expect a higher Caesareanfrequency among monitored labours, even ifmonitoring had no effect. 12 Where the indication(arrested labour) influences the use of theintervention (fetal monitoring) and is a risk factorfor the outcome (Caesarean section), there will bean imbalance in prognosis between the treatmentgroups being compared.Confounding by severity (or by prognosis) may beconsidered as a special form of confounding byindication. It occurs where the severity of theextraneous condition influences both theintroduction of the intervention and the outcomeunder study, 13 that is, any treatment reserved forthe most ill will be associated with the worstoutcomes (even where the treatment is effective).The underlying deterioration of the treateddisease predicts the use of a given treatment,which in turn becomes associated with theprogression of the disease. 14 For example, thefrequent use of β-agonists predicts death fromasthma because those with most severe asthma aremore likely to be prescribed β-agonists.Protopathic bias is a term coined by Horwitzand Feinstein 15 to describe situations where thefirst symptoms of a given outcome are the reasonfor treatment initiation: “Protopathic bias”occurs “when a pharmaceutical or othertherapeutic agent is inadvertently prescribed foran early manifestation of a disease that has not yetbeen diagnostically detected” (our emphasis). Forexample, a drug given for abdominal pain may bewrongly associated with hepatic injury, asabdominal pain may be one of the prodromalsymptoms. 13Unfortunately, in practice, the ‘subtle clues’prompting the selection of a particularintervention for a given patient are oftenunrecognised and unrecorded. When discussingthe use of observational databases, Byar 16 pointedout that “I have yet to see an analysis presentedtoday (or in my whole life for that matter) thathad really good information on why some patientsgot one treatment and others got another”.One approach to improving the design of nonrandomisedstudies is the ‘restricted cohort’ designfirst developed by Horwitz and Feinstein 15 and laterrefined by the same group. 17 This approach involvesrestricting the eligibility criteria of cohort studies tothose used in clinical trials, defining a ‘zero-time’point from which patients are followed up, andusing an approximation of intention-to-treatanalysis. They have demonstrated, for exampleusing a study of β-blocker therapy after acutemyocardial infarction, 17 that similar results could beobtained to those reported by RCTs. This approachdoes not appear to have been widely taken up, andthe original proposal has received some criticism. 3The degree to which non-random allocationmethods are susceptible to selection bias andtherefore may produce biased estimates of theeffect of treatment is not clearly understood,although it seems likely that the potential for biaswill vary between clinical areas. It is reasonable toexpect that the comparability of the groups interms of prognostic factors, and the extent towhich prognosis influences both selection fortreatment and treatment outcome will be ofparticular relevance. For example, in evaluationsof childhood vaccines there are few indicators ofprognosis that could be used to influenceallocation, so randomisation may not be necessary(although there are dangers that allocation inclusters could be confounded by exposure toinfectious disease, which means that in practicerandomisation is recommended). By contrast,when patient factors could have a strong influenceon allocation or where prognosis is strongly linkedto outcomes (such as in cancer treatment), thenrandomisation is likely to be extremely important.Other biases in non-randomisedstudiesNon-randomised studies are also susceptible toattrition, detection and performance bias. Forexample, attrition bias will occur if there are dropouts,detection bias if the assessment of outcomesis not standardised and blinded, and performancebias if there are errors and inconsistencies in theallocation, application and recording ofinterventions (Table 1). All of these biases can alsooccur in RCTs, but there is perhaps potential fortheir impact to be greater in non-randomisedstudies which are usually undertaken withoutprotocols specifying standardised interventions,outcome assessments and data recordingprocedures. A comprehensive assessment of thevalidity of both randomised and non-randomisedstudies requires the assessment of all dimensionsof potential bias.
Health Technology Assessment 2003; Vol. 7: No. 27Case-mix adjustment methodsIn the absence of information on factorsinfluencing allocation, the traditional solution toremoving selection bias in non-randomised studieshas been to attempt to control for knownprognostic factors, either by design and/or byanalysis. Alternative statistical techniques forremoving bias in non-randomised studies arebriefly described by D’Agostino and Kwan 1 andinclude matching, stratification, covarianceadjustment and the propensity score analysis(Box 2). However, all statistical techniques maketechnical assumptions (regression models typicallyassume that the relationship between theprognostic variable and the outcome is linear)and the degree to which they can adequatelyadjust for differences between groups isunclear.Moses 18 lists three factors necessary for successfuladjustment: (1) knowledge of which variables mustbe taken into account; (2) measuring thosevariables in each participant; and (3) using thosemeasurements appropriately to adjust thetreatment comparison. He further states that weare likely to fail on all three counts: 18“We often don’t understand the factors that causepeople’s disease to progress or not; even if we knewthose factors, we might find they had not beenmeasured. And if they were measured, the correct wayto use them in adjustment calls for a theoreticalunderstanding we seldom have.”BOX 2 Commonly used methods of statistical adjustmentStandardisationParticipants are analysed in groups (strata) which havesimilar characteristics, the overall effect being estimatedby averaging the effects seen in each of the groups.RegressionRelationships between prognostic factors and outcomeare estimated from the data in hand, and adjustmentscalculated for the difference in average values of theprognostic factor between the two groups. Linearregression (or covariance analysis) is used for continuousoutcomes, logistic regression for binary outcomes.Propensity scoresPropensity probabilities are calculated for each participantfrom the data set, estimating their chance of receivingtreatment according to their characteristics. Treatmenteffects are estimated either by comparing groups thathave similar propensity scores (using matching orstratification methods), or by calculating a regressionadjustment based on the difference in average propensityscore between the groups.Use of adjustment therefore assumes thatresearchers know which are the most importantprognostic factors and that these have beenappropriately measured. Further, it cannotaddress the problem of unknown orunmeasurable prognostic factors, which may playa particular role in confounding by indication.Full adjustment for confounding by indicationwould require information on prognostic factorsinfluencing both disease progression andtreatment choice. 14 Where there is an associationbetween prognosis and treatment choice, it wouldseem that correlates of prognosis can only warn ofa problem, not control for it. Furthermore, if thedegree to which the markers are correlated withprognosis is different in diseased and nondiseasedpersons then the magnitude (anddirection) of the resulting bias will beunpredictable in an overall analysis. 14Scope of this projectThis report contains the results of three systematicreviews and two empirical studies all relating tothe internal validity of non-randomised studies.The findings of these five studies will be ofimportance to researchers undertaking newstudies of healthcare evaluations and systematicreviews, and also healthcare professionals,consumers and policy makers looking to use theresults of randomised and non-randomised studiesto inform their decision-making.The first systematic review looks at existingevidence of bias in non-randomised studies,critically evaluating previous methodologicalstudies that have attempted to estimate andcharacterise differences in results between RCTsand non-randomised studies.Two further systematic reviews focus on the issueof quality assessment of non-randomised studies.The first identifies and evaluates tools that can beused to assess the quality of non-randomisedstudies. The second looks at ways that studyquality has been assessed and addressed insystematic reviews of healthcare interventions thathave included non-randomised studies.The two empirical investigations focus on the issueof selection bias in non-randomised studies. Thefirst investigates the size and behaviour ofselection bias in evaluations of two specific clinicalinterventions and the second assesses the degreeto which case-mix adjustment corrects forselection bias.5© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2: Health Technology Assessment 2003;
Page 3 and 4: Evaluating non-randomisedinterventi
Page 7: Health Technology Assessment 2003;
Page 23 and 24: © Queen’s Printer and Controller
Page 27 and 28: © Queen’s Printer and Controller
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 64 and 65: Empirical estimates of bias associa
Page 66 and 67:
Empirical estimates of bias associa
Page 68 and 69:
Page 70 and 71:
Page 72 and 73:
Page 74 and 75:
Page 76 and 77:
Empirical evaluation of the ability
Page 78 and 79:
Page 80 and 81:
Page 82 and 83:
70TABLE 22 Comparison of concurrent
Page 84 and 85:
Page 86 and 87:
74TABLE 26 Comparison of methods of
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
80TABLE 33 Hypothetical example dem
Page 94 and 95:
82TABLE 34 Hypothetical example dem
Page 96 and 97:
Page 98 and 99:
Page 100 and 101:
Discussion and conclusions88histori
Page 102 and 103:
Discussion and conclusions90For exa
Page 104 and 105:
Discussion and conclusionsNon-rando
Page 107 and 108:
Health Technology Assessment 2003;
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121:
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

Create successful ePaper yourself

Delete template?

Save as template?