Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

70TABLE 22 Comparison of concurrently and historically controlled studies with results of RCTs resampled from 14 regions within the IST analysed according to the number of significant differencesin baseline covariatesStudy design Maximum no. of significant Percentage of Average Ratio of Percentage of studies with statisticallybaseline differences studies meeting OR SD of log OR significant results (p < 0.05)(p < 0.05) for match a criterionBenefit Harm TotalRCT 0.89 9 2 11Non-randomised historical control Any number 100 0.88 1.3 15 4 190 or 1 or 2 79 0.89 1.3 14 4 180 or 1 55 0.89 1.2 12 3 150 23 0.92 1.2 12 3 15RCT 0.91 7 2 9Non-randomised concurrent control Any number 100 0.91 2.6 29 22 510 or 1 or 2 20 0.93 2.8 27 19 460 or 1 6 0.91 2.1 26 16 420 1 0.86 1.5 20 7 27a Baseline differences were considered for 10 variables: time since symptoms, consciousness, age, sex, presence of atrial fibrillation, stroke noticed on waking, systolic bloodpressure, infarct visible on CT scan, type of stroke and a score of eight neurological deficits.TABLE 23 Comparison of concurrently and historically controlled studies with results of RCTs resampled from eight regions within the ECST analysed according to the number of significantdifferences in baseline covariatesStudy design Maximum no. of significant Percentage of Average Ratio of Percentage of studies with statisticallybaseline differences studies meeting OR SD of log OR significant results (p < 0.05)(p < 0.05) for match a criterionBenefit Harm TotalRCT 1.23 4 12 16Non-randomised historical control Any number 100 1.06 1.0 6 10 160 or 1 or 2 86 1.01 1.0 7 9 160 or 1 57 0.93 1.0 8 7 150 18 0.81 0.9 9 3 12RCT 1.08 3 5 9Non-randomised concurrent control Any number 100 1.08 1.0 3 6 90 or 1 or 2 69 1.06 1.0 3 5 80 or 1 40 1.04 1.0 3 4 70 12 1.17 0.9 3 4 7Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection biasa Baseline differences were considered for eight variables: severity of presenting stroke, degree of stenosis, age, prophylactic aspirin use, angina, previous myocardial infarction,residual neurological signs and sex.
Health Technology Assessment 2003; Vol. 7: No. 27as the exponential of the mean of the log OR] andspread [the standard deviation of the observed logOR]. As in Chapter 6, ratios of the the averageORs of the RCTs and the average ORs of the nonrandomisedstudies (adjusted or unadjusted)quantify systematic bias, while ratios of their SDsindicate unpredictability in the bias. The likelyconclusions of the analyses were investigated byconsidering the percentage of studies for eachanalysis which reported statistically significant (p< 0.05) results, separately in the direction of harmand of benefit.ResultsExclusion of non-matching groupsTable 22 presents the results for analyses of the ISTaccording to the number of covariates thatdiffered significantly between treatment andcontrol groups. Table 23 presents comparableanalyses for the ECST.For the IST, 23% of the historically controlledstudies appeared to have comparable groups inthat they had no statistically significant differencesin baseline covariates (Table 22). There did appearto be a reduction in both systematic andunpredicatable dimensions of bias as the numberof differing covariates reduced. However, amongstudies with no significantly unbalanced covariatesthe results were still more variable than those fromthe corresponding RCTs, and there was still anexcess of statistically significant results.Table 23 shows that 18% of the historicallycontrolled non-randomised studies generatedfrom the ECST had no statistically significantdifferences in baseline covariates. A reverse trendwas noted with study results becoming morebiased with fewer significant differences inbaseline, contrary to the hypothesis thatcomparability at baseline predicts reliable results.Only 1% of concurrently controlled studies fromthe IST had no significant differences at baseline(Table 22), and reductions in excess variability wereless marked than for historically controlled studies.Further, the absence of statistically significantdifferences did not guarantee comparability, withspuriously statistically significant treatment effectsstill being common when treated and controlgroups appeared to match.As shown in Table 23, 12% of the concurrentlycontrolled non-randomised studies generatedfrom the ECST had no statistically significantdifferences in baseline covariates, but againsystematic bias was actually higher in thissubgroup of apparently comparable studies thanin the studies which had significant differences inat least one baseline covariates.Adjustment for ‘naturally’ occurringbiasesSystematic bias originating from historicallycontrolled studiesThe clearest example of systematic bias wasobserved in historically controlled studies in theECST data, where the average OR estimate was1.06 compared with 1.23 in the RCTs, grosslyunderestimating the harmfulness of treatment(see Chapter 6). The seven case-mix adjustmentmethods were applied to each of the historicallycontrolled studies to investigate the degree towhich the case-mix methods could adjust for thisbias. The results are presented in Table 24 andFigure 15. The adjusted results from six of theseven methods were on average more biased thanthe unadjusted results: only the full logisticregression model appeared to reduce bias. Allmethods also increased variability(unpredictability in bias), the greatest increasebeing with the full logistic regression. Thisincrease in the width of distributions of results isdiscernible in Figure 15.The use of adjustment methods inflates thestandard error of the estimate of a treatmenteffect. The use of propensity score matching led inaddition to a reduction in sample size (as 45% ofparticipants were discarded), further reducingpower. Thus, although many of the adjustedestimates were on average more biased than theunadjusted estimates, only logistic regressionmethods had markedly increased spurioussignificance rates.Systematic bias was also observed in thehistorically controlled studies generated from theIST, although it differed between regions. Overalla small systematic bias was noted in theaggregated results (Table 25). Logistic regressionmodelling showed a similar pattern of behaviouras for the ECST comparison, increasing bothaverage bias and the variability of results.Propensity score methods slightly overadjusted forthe bias, but gave results closest to the RCTresults. Spurious statistical significance ratesdecreased slightly with logistic regression and werecompletely removed by propensity score methods.The case-mix adjusted results for the historicallycontrolled studies from the IST analysis are71© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2:
Health Technology Assessment 2003;
Page 3 and 4:
Evaluating non-randomisedinterventi
Page 5 and 6:
Page 7:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17:
Page 21 and 22:
Page 23 and 24:
© Queen’s Printer and Controller
Page 25 and 26:
Page 27 and 28:
Page 29 and 30:
Page 31 and 32: Health Technology Assessment 2003;
Page 33: Health Technology Assessment 2003;
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 76 and 77: Empirical evaluation of the ability
Page 86 and 87: 74TABLE 26 Comparison of methods of
Page 92 and 93: 80TABLE 33 Hypothetical example dem
Page 94 and 95: 82TABLE 34 Hypothetical example dem
Page 100 and 101: Discussion and conclusions88histori
Page 102 and 103: Discussion and conclusions90For exa
Page 104 and 105: Discussion and conclusionsNon-rando
Page 121: Health Technology Assessment 2003;
Page 124 and 125: Appendix 1data)) or (non-random$ or
Page 126 and 127: Appendix 2AuthorYearENDARESourcePub
Page 128 and 129: Appendix 2Author:Accession No:Endno
Page 130 and 131: Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?