Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

Discussion and conclusions90For example, the biases observed in our empiricalinvestigations based on cohort designs could notbe removed using case-mix adjustment methods,even though we could describe the process bywhich allocation occurred (Chapter 7). This wasdespite having at our disposal complete data onseveral highly prognostic variables. Our principalfailure resulted from being unable to describe howallocation was linked to prognostic factors in amanner that could be adjusted for in a statisticalanalysis. Moses 18 stated that there are threerequirements for successful case-mix adjustment:(1) knowledge of which variables must be takeninto account, (2) measuring those variables on eachparticipant and (3) using those measurementsappropriately to adjust the treatment comparison.He suggested that researchers are likely to fail at allthree. Although it is well known that failure toidentify all relevant variables will lead to underadjustment(residual confounding), our evaluationshave suggested that failing to measure thecorrect variables properly and failing to modeltheir effects appropriately can lead to spuriousconclusions.The use of case-mix adjustment analyses shouldtherefore not be regarded as a guarantee that astudy is unbiased, or that the observed differencesin case-mix have been adjusted for. Investigatorsusing case-mix adjustment methodologies,especially logistic regression models, should drawconclusions cautiously, and probably regardadjusted analyses as exploratory rather thanconfirmatory. Our observations require furtherassessment and confirmation from investigationsin additional data sets to explore further themechanisms that are causing these failures.Nonetheless, they raise serious concern about theroutine promotion and use of case-mix adjustmentmethods for the analysis of non-randomisedstudies. We do not propose that case-mixadjustment be abandoned, but recommend thatgreater scepticism be applied in the interpretationof results.What are the implications for thoseproducing, reviewing and using nonrandomisedstudies?An investigator planning to undertake a nonrandomisedstudy should first make certain thatan RCT cannot be undertaken. 18 The advantagesof concealed randomised allocation are so greatcompared with the inadequacies that we havequantified for non-randomised studies that itshould be with the greatest reluctance that aninvestigator concludes that randomisation is notpossible. The ability to eradicate bias at thedesign stage is crucial to establishing thevalidity of a study. In particular, investigatorsshould not assume that statistical methods canbe used reliably to compensate for biasesintroduced through suboptimal allocationmethods.A prospective non-randomised study should beundertaken according to a protocol that iscarefully followed to ensure consistent inclusioncriteria, that all relevant factors are measuredaccurately for each participant and thatparticipants are all monitored in a standardmanner and blinded to treatment if possible. Insome situations it may even be possible to matchprospectively treated and control patients onimportant prognostic factors. 171 Byar pointed outthat the one piece of information required forsuccessful adjustment is knowledge of the reasonswhy each patient received their particulartreatment. 16 This information is usually notrecorded in non-randomised studies, and how itshould be measured is a real challenge worthy ofresearch. 18 As a minimum, recording details ofreasons for allocations of particular interventionsshould allow the subset of patients to be identifiedwho are not considered suitable for bothtreatments, and should not be included inanalyses. In addition, all prognostic variablesshould be measured in such a way as to allowmeasurement error, misclassification and withinpersonvariability to be assessed. Some authorshave attempted to acquire reliable risk assessmentsby retrospective case-note review, 172 but obtainingthese data prospectively has many attractions.Retrospective studies cannot use consistentinclusion criteria, or ensure that data arecomplete and consistently recorded. Phillipsand Davey Smith have pointed out that it isprobably more worthwhile to put effort intoundertaking a small observational study to ahigh standard than in obtaining poor qualityand inadequate data on a large number ofparticipants. 173 However, despite all these efforts,the possibility of residual confounding will not beremoved.Several authors have suggested additionalstrategies that investigators might consideras a way of identifying hidden bias. Rosenbaumproposed the routine use of extra controlgroups and the inclusion of additionaloutcomes that are known not to be alteredby the treatment as further checks oncomparability. 145,152 Several authors haveemphasised the use of sensitivity analyses toquantify the strength of confounding in a missing
Health Technology Assessment 2003; Vol. 7: No. 27covariate that would be needed to nullify anobserved effect. 145,174 However, none of thesemethods can correct for bias.Investigators undertaking systematic reviews ofeffectiveness should include the results of nonrandomisedstudies with discretion. If results fromgood-quality RCTs are available, there seems to beno justification for additionally considering nonrandomisedstudies. If non-randomised studies areto be included in a review, their quality needs tobe carefully assessed to evaluate the likelihood ofbias. Quality assessment should pay particularattention to the description of the allocationmechanisms and the demonstration of there beingcomparability in all important prognostic factorsat baseline. Although we have not identified aquality assessment tool that met all ourrequirements, the six best tools could all beadapted for use in a systematic review. Theconclusions of a review should take into accountthe extra uncertainty associated with the results ofnon-randomised studies.The results of non-randomised studies should betreated with a healthy degree of scepticism.Healthcare decision-makers should be cautious notto over-interpret results from non-randomisedstudies. Importantly, checking that treated andcontrol groups appear comparable does notguarantee freedom from bias, and it should neverbe assumed that case-mix adjustment methods canfully correct for observed differences betweengroups. The uncertainty in the result of a nonrandomisedstudy is not properly summarised inthe confidence interval for the overall effect: ouranalyses have shown that the true uncertainty inthe results of non-randomised studies may be 10times greater.ConclusionsNon-randomised studies are sometimes but notalways biased:●●The results of non-randomised studies can differfrom the results of RCTs of the same intervention.All other issues remaining equal, lack ofrandomisation introduces bias into theassessment of treatment effects. The bias mayhave two components. It may be systematic andappear on average to act in a particulardirection if the non-random allocationmechanism leads to a consistent difference incase-mix. However, if the allocation mechanismcan lead to haphazard differences in case-mix,●the bias can act in either direction, increasinguncertainty in outcome in ways that cannot bepredicted. The extent of systematic bias andincreased uncertainty varies according to thetype of non-randomised comparison andclinical context.Meta-epidemiological techniques tend not toprovide useful information on the sources ofand degrees of bias in non-randomised studiesowing to the existence of meta-confounding andlack of systematic or predictable bias in theresults of non-randomised studies.Statistical methods of analysis cannot properlycorrect for inadequacies of study design:●Case-mix comparability and standard methodsof case-mix adjustment do not guarantee theremoval of bias. Residual confounding may behigh even when good prognostic data areavailable, and in some situations adjustedresults may appear more biased thanunadjusted results.Systematic reviews of effectiveness often do notadequately assess the quality of non-randomisedstudies:●●Quality assessment has not routinely beenundertaken in systematic reviews of effectivenessthat include non-randomised studies. Whenstudy quality has been investigated, there isvariability in the tools and quality criteria thathave been used and there has been no consistentpattern between quality and review findings.Not all reviews that have assessed quality haveconsidered the findings when synthesising studyresults and drawing conclusions.Although many quality assessment tools existand have been used for appraising nonrandomisedstudies, few are suitable for thistask as they omit key domains of qualityassessment (assessment of allocationmechanisms, attempts to achieve case-mixcomparability by design, identification ofconfounding factors, use of case-mix adjustmentmethods). Few were developed usingappropriate methodological procedures.Fourteen tools were identified which appear tohave reasonable coverage of core domains ofinternal validity, six of which were consideredpotentially suitable for use in systematic reviewsof non-randomised studies. All six wouldrequire modification to cover adequately the keyissues in non-randomised studies (of identifyingprognostic factors and accounting for them inthe design and analysis).91© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2:
Health Technology Assessment 2003;
Page 3 and 4:
Evaluating non-randomisedinterventi
Page 5 and 6:
Page 7:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17:
Page 21 and 22:
Page 23 and 24:
© Queen’s Printer and Controller
Page 25 and 26:
Page 27 and 28:
Page 29 and 30:
Page 31 and 32:
Page 33:
Page 36 and 37:
Evaluation of checklists and scales
Page 38 and 39:
Page 40 and 41:
Page 42 and 43:
Page 44 and 45:
32TABLE 8 Details of top 60 quality
Page 46 and 47:
Page 48 and 49:
Page 50 and 51:
38TABLE 10 Other domains: reporting
Page 52 and 53: Evaluation of checklists and scales
Page 54 and 55: Evaluation of checklists and scales
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 76 and 77: Empirical evaluation of the ability
Page 82 and 83: 70TABLE 22 Comparison of concurrent
Page 86 and 87: 74TABLE 26 Comparison of methods of
Page 92 and 93: 80TABLE 33 Hypothetical example dem
Page 94 and 95: 82TABLE 34 Hypothetical example dem
Page 100 and 101: Discussion and conclusions88histori
Page 104 and 105: Discussion and conclusionsNon-rando
Page 107 and 108: Health Technology Assessment 2003;
Page 121: Health Technology Assessment 2003;
Page 124 and 125: Appendix 1data)) or (non-random$ or
Page 126 and 127: Appendix 2AuthorYearENDARESourcePub
Page 128 and 129: Appendix 2Author:Accession No:Endno
Page 130 and 131: Appendix 20 0 00Additional outcomes
Page 132 and 133: Appendix 2Endnote NoWas CMA conside
Page 134 and 135: 122AuthorOrigin aModified toolTool
Page 148 and 149: Appendix 4136DuRant, 1994 99The typ
Page 153 and 154:
Page 155 and 156:
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

Create successful ePaper yourself

Delete template?

Save as template?