Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

Empirical estimates of bias associated with non-random allocation58results between non-randomised and randomisedstudies. The bias was observed for the ECSThistorically controlled comparisons leading tooverestimates of the benefit of carotid surgery,both for individual regions and when the resultswere aggregated across regions. This pattern isconsistent with the conclusions of Sacks andcolleagues, 27 who noted in their review ofhistorically controlled studies for six medicalinterventions that “biases in patient selectionmay irretrievably weight the outcome ofhistorically controlled studies in favour of newtherapies”.Systematic biases were also noted in some of thehistorically controlled studies in the individualregions in the IST analysis, but here they wereseen to vary in direction and magnitude,sometimes overestimating benefit and sometimesoverestimating harm.Systematic bias in historically controlled studiesarises from there being time trends in theaverage outcomes of participants in a study,regardless of which treatment they receive.Details of the outcomes and characteristicsof the participants in the ECST are presentedin Tables 41 and 42 in Appendix 8. For fiveregions there was a reduction in the adverseevent rate of between 1 and 7% (averaged acrossboth treatment and control) between the trialperiods, whereas for three regions there was anincrease of between 1 and 14%. The changewas statistically significant (p < 0.01) in oneregion.How do such trends arise? There are a limitednumber of options: they must arise throughvariation over time in the case-mix, and henceprognosis, of participants recruited to the trial (asproposed by Sacks and colleagues 27 ), throughdifferences in other healthcare interventions thatthe participants receive or through changingassessments of outcome. These variations maythemselves be haphazard or due to systematicmechanisms (such as changes in patient referraland recruitment or in patient management). Someof these potential causes may be measured, such asbaseline risk factors, but many may go unnoticedand are not assessed.Tables 39 and 42 in Appendix 8 show summariesof the distribution of important baseline riskfactors for IST and ECST, respectively. For bothtrials there were differences in the risk factors ofparticipants between the first and second halves ofthe trial, although the patterns of these differenceswere not consistent between regions, and it is notimmediately obvious how they relate to differencesin outcome. It seems likely that the differencesoccur in part due to unmeasured changes withinthe trials, but that there may also be differentmechanisms causing systematic bias in differentregions.Why should there be a time trend in outcome inthe ECST? Patients were only entered into thetrial when an investigator judged that in the caseof the individual patient there was uncertainty asto whether surgery would be beneficial orharmful. One possibility is therefore thatthroughout the very long recruitment period(12.5 years) investigators joined or left the trialwho had systematically different opinions on whowas suitable for randomisation. Six of the eightregions showed significant reductions (p < 0.05)in the proportion of patients recruited with
Health Technology Assessment 2003; Vol. 7: No. 27Unpredictability in biasWhen bias acts unpredictably, it will sometimeslead to an overestimation and sometimes tounderestimation of an effect. Although thesebiases may on average ‘cancel out’ across a set ofstudies such that no difference is observed inaverage ORs, the biases will still affect the resultsof individual studies. The presence of systematicbias may therefore be missed if the comparison ofresults is restricted to a comparison of averagevalues, as was done in five of the eight previousreviews summarised in Chapter 3. 25–28,32Unpredictable over- and underestimation willincrease the variability (or heterogeneity) of theresults of a set of studies. In the concurrentcomparisons such an increase in variability(measured by the standard deviation) wasobserved for the IST (Table 15), even though theaverage treatment effects in the concurrentlycontrolled and randomised studies were thesame. A similar pattern was observed forhistorically controlled studies generated from theIST when the haphazard within-region timetrends were aggregated in the overall analysis(Table 16).How do these biases occur, and how do they differfrom the variability seen between RCTs? Variabilityalways occurs between the results of multipleRCTs. The principal reason is the ‘play of chance’or sampling variation. A treatment effect observedin a particular RCT is unlikely to be the preciseeffect of the intervention. For example, randomlydividing the study sample into two does notguarantee that the groups are identical in allrespects, and the differences that do exist in casemixwill lead to either under- or overestimates ofthe treatment effect in an individual trial. We donot normally talk about these differences asbiases, but rather as uncertainties. We know thedistribution with which under-and overestimatesarise in RCTs, enabling us to draw correctinferences within specified degrees of certainty. Wecannot identify whether a particular trial isaffected by such bias, but we can calculate boundswithin which we are reasonably sure possible biasis encompassed, which we term confidenceintervals. Importantly, we know that the possibledifferences between the groups due to samplingvariation (and hence confidence intervals) reducewith increasing sample size.The extra variability we see in the nonrandomisedstudies arises in a similar but moretroubling manner. Rather than randomly dividinga single group of individuals, we start with twodifferent groups of individuals. We therefore startwith differences between the groups in measurableand unmeasurable factors. These potentiallyinclude differences in case-mix, additionaltreatments and methods of assessment ofoutcome. Importantly, in addition to not beingable to identify all these differences, we may notknow in which way many of the factors act, so thatthere is overall uncertainty as to whether they willcause under- or overestimates of the treatmenteffect. Sampling from these populationsintroduces the same sampling variation as in theRCT. While we can estimate the impact of thesampling variation (and calculate standardconfidence intervals), there is no mathematicalway of knowing how pre-existing differencesbetween the groups behave. It is therefore notpossible to include an allowance in the confidenceinterval for a single study that accounts for theextra uncertainty introduced through unsystematicbias. As we cannot mathematically allow for thisvariation when drawing conclusions, it isappropriate to call such extra variation ‘bias’ eventhough it is ‘uncertain’. In contrast to samplingvariation, the extra uncertainty is independent ofsample size as it is a feature of the pre-existingdifferences between the two populations fromwhich the samples were drawn.Our resampling studies provide a uniqueopportunity to calculate the distribution of thisextra uncertainty for the specific situations studiedin the IST and ECST by calculating the increase invariance seen with non-randomised concurrentlycontrolled studies compared with RCTs. Thiscomputation is possible as we ensured that foreach study the RCTs are the same size as theconcurrent comparisons, such that the differencesin variability cannot be explained by differences insampling variability. The results of thesecomputations are given in Table 19. The extravariance in log OR was 0.61 for regional ISTcomparisons, 0.57 for UK city IST comparisonsand 0.01 for regional ECST comparisons. Giventhese estimates, it is possible to calculate newadapted confidence intervals for these studies thatallow for these potential uncertain biases inaddition to sampling variation. They areexpressed in Table 19 as multiplicative increases inthe width of the standard confidence intervals. Assampling variability decreases with increasingsample size but the unsystematic bias remainsconstant, the ratio of the extra allowance in thewidth of the confidence interval due tounsystematic bias increases with sample size. Theratios presented in Table 19 reveal that standardconfidence intervals for many non-randomised59© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2:
Health Technology Assessment 2003;
Page 3 and 4:
Evaluating non-randomisedinterventi
Page 5 and 6:
Page 7:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17:
Page 21 and 22: Health Technology Assessment 2003;
Page 23 and 24: © Queen’s Printer and Controller
Page 27 and 28: © Queen’s Printer and Controller
Page 33: Health Technology Assessment 2003;
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 76 and 77: Empirical evaluation of the ability
Page 82 and 83: 70TABLE 22 Comparison of concurrent
Page 86 and 87: 74TABLE 26 Comparison of methods of
Page 92 and 93: 80TABLE 33 Hypothetical example dem
Page 94 and 95: 82TABLE 34 Hypothetical example dem
Page 100 and 101: Discussion and conclusions88histori
Page 102 and 103: Discussion and conclusions90For exa
Page 104 and 105: Discussion and conclusionsNon-rando
Page 121:
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?