Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

Executive summaryUse of quality assessment in systematicreviews of non-randomisedstudiesOf 511 systematic reviews that included nonrandomisedstudies, only 169 (33%) assessed studyquality. Many used quality assessment toolsdesigned for RCTs or developed by the authorsthemselves, and did not include key qualitycriteria relevant to non-randomised studies. Sixtyninereviews investigated the impact of quality onstudy results in a quantitative manner.Empirical estimates of bias associatedwith non-random allocationThe bias introduced by non-random allocation wasnoted to have two components. First, the biascould lead to consistent over- or underestimationsof treatment effects. This occurred for historicalcontrols, the direction of bias depending on timetrends in the case-mix of participants recruited tothe study. Second, the bias increased variation inresults for both historical and concurrent controls,owing to haphazard differences in case-mixbetween groups. The biases were large enough tolead studies falsely to conclude significant findingsof benefit or harm.Empirical evaluation of case-mixadjustment methodsFour strategies for case-mix adjustment wereevaluated: none adequately adjusted for bias inhistorically and concurrently controlled studies.Logistic regression on average increased bias.Propensity score methods performed better, butwere not satisfactory in most situations. Detailedinvestigation revealed that adequate adjustmentcan only be achieved in the unrealistic situationwhen selection depends on a single factor.Omission of important confounding factors canexplain underadjustment. Correlatedmisclassifications and measurement error inconfounding variables may explain the observedincrease in bias with logistic regression, as maydifferences between conditional and unconditionalodds ratio estimates of treatment effects.ConclusionsResults of non-randomised studies sometimes, butnot always, differ from results of randomisedstudies of the same intervention. Non-randomisedstudies may still give seriously misleading resultswhen treated and control groups appear similar inkey prognostic factors. Standard methods of casemixadjustment do not guarantee removal of bias.Residual confounding may be high even whengood prognostic data are available, and in somesituations adjusted results may appear more biasedthan unadjusted results.Although many quality assessment tools exist andhave been used for appraising non-randomisedstudies, most omit key quality domains. Six toolswere considered potentially suitable for use insystematic reviews, but each requires revision tocover all relevant quality domains.Healthcare policies based upon non-randomisedstudies or systematic reviews of non-randomisedstudies may need re-evaluation if the uncertaintyin the true evidence base was not fully appreciatedwhen policies were made.The inability of case-mix adjustment methods tocompensate for selection bias and our inability toidentify non-randomised studies which are free ofselection bias indicate that non-randomisedstudies should only be undertaken when RCTs areinfeasible or unethical.Recommendations for furtherresearch●●●●●The resampling methodology utilised hereshould be applied in other clinical areas toascertain whether the biases we describe aretypical.Efforts should be focused on developing qualityassessment tools for non-randomised studies,possibly by refining existing tools.Research should consider how qualityassessment of non-randomised studies can beincorporated into reviews, and the implicationsof individual quality features for interpretationof a review’s results.Reasons for the apparent failure of case-mixadjustment methods should be furtherinvestigated, including assessments of thegeneralisability of our results to risk assessmentsand epidemiological studies, and differencesbetween conditional and unconditionalestimates of treatment effects.The role of the propensity score should befurther evaluated, and computer macros madeavailable for its application.x
Health Technology Assessment 2003; Vol. 7: No. 27Chapter 1IntroductionThe ultimate goal of the evaluation ofhealthcare interventions is to produce a validestimate of effectiveness, in terms of both internaland external validity. Internal validity concerns theextent to which the results of a study can bereliably attributed to the intervention underevaluation, whereas external validity concerns theextent to which a study’s results can be generalisedbeyond the given study context.The randomised controlled trial (RCT) is widelyregarded as the design of choice for theassessment of the effectiveness of healthcareinterventions. The main benefit of the RCT is theuse of a randomisation procedure that, whenproperly implemented, ensures that the allocationof any participant to one treatment or anothercannot be predicted. The randomisation processmakes the comparison groups equal with respectto both known and unknown prognostic factors atbaseline, apart from chance bias. 1 RCTs also tendto benefit from so-called ‘inherited properties’,which generally mark them out as higher qualitystudies. These properties include the fact that theyare prospective studies, with written protocolsspecifying, and thus standardising, importantaspects of patient enrolment, treatment,observation and analysis. 2 RCTs are also morelikely to employ specific measures to reduce orremove bias, such as blinded outcome assessment.There are instances where non-randomised studieshave either been sufficient to demonstrateeffectiveness or they appear to have arrived atresults similar to those of RCTs. However, whererandomisation is possible, most agree that theRCT should be the preferred method ofevaluating effectiveness. 2–4 The risks of relyingsolely on non-randomised evidence include failingto convince some people of the validity of theresult, or successfully convincing others of anincorrect result. 3Nevertheless, several scenarios remain underwhich an RCT may be unnecessary, inappropriate,impossible or inadequate. 5 Examples include theassessment of rare side-effects of treatments, somepreventive interventions and policy changes.Furthermore, there must be hundreds of examplesof interventions for which RCTs would be possiblebut have not yet been carried out, leaving themedical and policy community to rely on nonrandomisedevidence. It is therefore essential tohave an understanding of the biases that mayinfluence non-randomised studies.Types of non-randomised studiesA taxonomy of study designs that may be used toassess the effectiveness of an intervention isprovided in Box 1. However, there is inconsistentuse of nomenclature when describing nonrandomisedstudies, and other taxonomies mayapply different definitions to the same studydesigns. To attempt to avoid the problems ofinconsistent terminology, six features can beidentified that differentiate between these studies.First, some studies make comparisons betweengroups, whilst some simply describe outcomes in asingle group (e.g. case series). Second, thecomparative designs differ in the way thatparticipants are allocated to groups, varying fromthe use of randomisation (RCTs), quasirandomisation,geographical or temporal factors(cohort studies), the decisions of healthcareprofessionals (clinical database cohorts), to theidentification of groups with specific outcomes(case–control studies). Third, studies differ in thedegree to which they are prospective (andtherefore planned) or retrospective, for matterssuch as the recruitment of participants, collectionof baseline data, collection of outcome data andgeneration of hypotheses. Fourth, the methodused to investigate comparability of the groupsvaries: in RCTs no investigation is necessary(although it is often carried out), in controlledbefore-and-after designs baseline outcomemeasurements are used, and in cohort andcase–control studies investigation of confoundersis required. Fifth, studies differ in the level atwhich the intervention is applied: sometimes it isallocated to individuals, other times to groups orclusters.Finally, some studies are classified as experimentalwhereas others are observational. In experimentalstudies the study investigator has some degree ofcontrol over the allocation of interventions. Mostimportantly, he/she has control over the allocation1© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2: Health Technology Assessment 2003;
Page 3 and 4: Evaluating non-randomisedinterventi
Page 7: Health Technology Assessment 2003;
Page 23 and 24: © Queen’s Printer and Controller
Page 27 and 28: © Queen’s Printer and Controller
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63:
Empirical estimates of bias associa
Page 64 and 65:
Page 66 and 67:
Page 68 and 69:
Page 70 and 71:
Page 72 and 73:
Page 74 and 75:
Page 76 and 77:
Empirical evaluation of the ability
Page 78 and 79:
Page 80 and 81:
Page 82 and 83:
70TABLE 22 Comparison of concurrent
Page 84 and 85:
Page 86 and 87:
74TABLE 26 Comparison of methods of
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
80TABLE 33 Hypothetical example dem
Page 94 and 95:
82TABLE 34 Hypothetical example dem
Page 96 and 97:
Page 98 and 99:
Page 100 and 101:
Discussion and conclusions88histori
Page 102 and 103:
Discussion and conclusions90For exa
Page 104 and 105:
Discussion and conclusionsNon-rando
Page 107 and 108:
Health Technology Assessment 2003;
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121:
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?