Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

Empirical estimates of bias associated with non-random allocationConclusionsIndividual non-randomised studies may haveseriously biased results when either historical orconcurrent controls are used. Biases may belarge enough to mask both small and moderatetreatment effects, to the extent that nonrandomisedstudies may observe statisticallysignificant effects acting in the wrongdirection.While the use of historical controls may frequentlylead to overestimates of effectiveness of theexperimental treatment, this depends on theunderlying time trend of improving outcomes inthe patients being studied. Such a trend may notalways apply, however, especially when the casemixof those being considered for treatment alsochanges over time.Geographical variations in the provision of careand in the case-mix of patients being consideredfor treatment will lead to bias in studies usinggeographical concurrent controls. Differences incase-mix between treatment and control groupsare likely to be haphazard, such that the size andmagnitude of the biases may be unpredictable.The magnitude and nature of biases may differconsiderably between clinical situations.While results of RCTs and non-randomised studiesmay appear to agree on average across manystudies, this does not indicate that non-randomisedstudies are reliable, as individually they are affectedby additional unpredictability. Failure to recognisethis aspect may have led previous reviews falsely tounderestimate the bias in non-randomised studieswhen they compared average results.62
Health Technology Assessment 2003; Vol. 7: No. 27Chapter 7Empirical evaluation of the ability of case-mixadjustment methodologies to control forselection biasIntroductionCase-mix (or risk) adjustment methods are usedwidely throughout healthcare research to enablecomparisons to be made between groups that arenot directly comparable owing to differences inprognostic (or confounding) factors. For example,comparisons between outcomes from differenthospitals are often confounded by the severity ofthe patients that they treat, and it is recommendedthat outcomes should be compared only whendifferences in case-mix (severity) are adjustedfor. 136,137 The philosophy of case-mix adjustmentis to ‘level the playing field’ during analysis,enabling a comparison of like-with-like to be madeat the point of analysis, even if comparable groupscould not be generated by the study design. Casemixadjustment is an attempt to achieve byanalysis what could not be done (or was not done)by the design.Consideration of case-mix is routinelyrecommended in the analysis of non-randomisedstudies. Guides for assessing the validity of nonrandomisedstudies for both therapy and harmrecommend readers to assess first whether‘investigators demonstrate similarity in all knowndeterminants of outcome’ and, if not, whetherthey ‘adjust for differences in analysis’. 138,139 Manyepidemiological and biostatistical texts advocateadjustment for differences in baseline covariateswhen they are observed, both in non-randomisedcontrolled studies and even in RCTs. 140–142We will focus on comparisons between two groups,which we refer to as treatment (or experimental)and control. In principle, many of the methodsdiscussed extend to multiple armed trials. Fourapproaches to dealing with differences in case-mixin non-randomised studies are commonlyencountered in the medical literature:1. Comparison of baseline characteristics.The baseline clinical and demographiccharacteristics of the two groups are comparedto ascertain whether differences in case-mixexist and hence determine the certainty withwhich the observed difference can be attributedto the intervention and not to confoundingfactors. This method does not attempt to adjustfor differences in case-mix but simply todiscover whether there is evidence ofconfounding and make inferences accordingly.The presence of baseline differences is usuallydetermined by tests of statistical significance,although such tests do not relate directly tocomparability. 1432. Standardised or stratified analyses. Studyparticipants are divided into groups (strata)with the same characteristics. Stratified analyseswork by making ‘like-with-like’ comparisonswithin each of these groups. The overalltreatment effect is calculated by computing aweighted average of the within-strata estimatesof the treatment effect. The most popularmethod is that of Mantel and Haenszel. 144Stratification is best used when there are onlyone or two baseline characteristics, and isfrequently used in epidemiological research tostandardise for differences in age and/or sex.3. Multiple regression. Regression models (linearregression if the outcome is continuous,logistic regression if it is binary and Coxmodels if censoring occurs) estimate how eachprognostic factor relates to outcome. When thecomparison between the groups is made,adjustments are added to or subtracted fromthe estimated treatment effect to account forthe impact of differences in each of thebaseline covariates according to their estimatedrelationship with outcome. The two stages ofthe process happen simultaneously, so that theresults depend on the correlations betweenbaseline characteristics and treatmentallocation. Stepwise regression procedures arecommonly used to identify covariates that aresignificantly related to outcome, and work in asequential manner such that adjustments areonly made for the subset of covariates thoughtto matter.4. Propensity score methods. Propensity scoresare the least familiar method. Whereas63© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2:
Health Technology Assessment 2003;
Page 3 and 4:
Evaluating non-randomisedinterventi
Page 5 and 6:
Page 7:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17:
Page 21 and 22:
Page 23 and 24: © Queen’s Printer and Controller
Page 25 and 26: Health Technology Assessment 2003;
Page 27 and 28: © Queen’s Printer and Controller
Page 33: Health Technology Assessment 2003;
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 76 and 77: Empirical evaluation of the ability
Page 82 and 83: 70TABLE 22 Comparison of concurrent
Page 86 and 87: 74TABLE 26 Comparison of methods of
Page 92 and 93: 80TABLE 33 Hypothetical example dem
Page 94 and 95: 82TABLE 34 Hypothetical example dem
Page 100 and 101: Discussion and conclusions88histori
Page 102 and 103: Discussion and conclusions90For exa
Page 104 and 105: Discussion and conclusionsNon-rando
Page 121: Health Technology Assessment 2003;
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?