Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

Review of empirical comparisons of the results of randomised and non-randomised studies18Lipsey and Wilson 35 and Wilson and Lipsey 36Lipsey and Wilson searched for all meta-analysesof psychological interventions, broadly defined astreatments whose intention was to inducepsychological change (whether emotional,attitudinal, cognitive or behavioural). Evaluationsof individual components of interventions andbroad interventional policies or organisationalarrangements were excluded. Searches ofpsychology and sociology databases supported bymanual searches identified a total of 302 metaanalyses,76 of which contained both randomisedand non-randomised comparative studies. Resultswere analysed in two ways. First, the average effectsizes of randomised and non-randomised studieswere computed across the 74 reviews, and averageeffects were noted to be very slightly smaller fornon-randomised than randomised studies. Second(and more usefully) the difference in effect sizesbetween randomised and non-randomised studieswithin each of the reviews was computed andplotted. This revealed both large over- andunderestimates with non-randomised studies,differences in effect sizes ranging from –0.60 to+0.77 standard deviations.Studies excluded from the reviewThree commonly cited studies were excluded fromour review. 37–39 Although these studies madecomparisons between the results of randomisedand non-randomised studies across manyinterventions, they did not match RCTs and nonrandomisedstudies according to the intervention.Although they provide some information about theaverage findings of selected randomised and nonrandomisedstudies, they did not consider whetherthere are differences in results of RCTs and nonrandomisedstudies of the same intervention.Findings of the eight reviewsThe eight reviews have drawn conflictingconclusions. Five of the eight reviews concludedthat there are differences between the results ofrandomised and non-randomised studies in manybut not all clinical areas, but without there being aconsistent pattern indicating systematicbias. 25,26,28,34,35 One of the eight reviews found anoverestimation of effects in all areas studied. 27 Thefinal two concluded that the results of randomisedand non-randomised studies were ‘remarkablysimilar’. 32,33Of the two reviews that considered the relativevariability of randomised and non-randomisedresults, one concluded that RCTs were moreconsistent 34 and the other that they were lessconsistent. 33The two studies that investigated the impact ofcase-mix adjustment were in agreement, bothnoting that adjustment did not necessarily reducediscordance between randomised and nonrandomisedfindings. 25,27Critical evaluation of reviewsThe discrepancies in the conclusions of the eightreviews may in part be explained by variations intheir methods and rigour, so that they had varyingsusceptibility to bias. We consider the weaknessesin these reviews under four headings.Was the identification of included studiesunlikely to be biased?The studies used in all the reviews represent onlya very small portion of all randomised andobservational research. From Table 3, it is clear thatthe seven reviews in medical areas were each onlybased on a subset of known comparisons ofrandomised and non-randomised evidence. Eventhe largest review 34 did not include allcomparisons identified in previous reviews.Correspondence has also cited several otherexamples of treatment comparisons where thereare disagreements between observational andrandomised studies. 40,41More important, could the comparisons selectedfor these meta-epidemiological reviews be apotentially biased sample? There are two levels atwhich publication bias can act in these evaluations:(a) selective publication of primary studies, whichwill affect all reviews, and (b) selective publicationof meta-analyses of these studies, which will affectreviews restricted to secondary publications.Evaluations of publication bias have noteddifferences in the frequency of primary publicationof randomised and observational studies, althoughthe direction and magnitude of the differencesvary between evaluations and the relationship tostatistical significance is not known. 42 Similarly,the decisions made concerning the publication ofmeta-analyses that include non-randomisedstudies are likely to be influenced by the results ofexisting randomised controlled trials. TheConcato review 33 may be the most susceptible topublication bias as it restricted study selection tometa-analyses combining randomised orobservational results published in five generalmedical journals: Annals of Internal Medicine, BritishMedical Journal (BMJ), Journal of the AmericanMedical Association (JAMA), New England Journal ofMedicine (NEJM) and The Lancet. Therefore, theonly studies eligible for inclusion were those whereboth authors and top journal editors had alreadydecided that it was sensible to synthesise the
Health Technology Assessment 2003; Vol. 7: No. 27results of both observational and randomisedstudies. It seems highly likely that these decisionsmay relate to the similarity of the results of studiesof different designs.Did the RCTs and non-randomised studiesrecruit similar participants, use similarinterventions and measure similar outcomes?Discrepancies between the results of observationaland randomised studies may be confounded bydifferences in the selection and evaluation ofpatient groups, in the formulation and delivery oftreatments, in the use of placebos and othercomparative treatments and in the methods usedto maintain follow-up and record outcomes. Formany interventions there may also be temporalconfounding of study types, non-randomisedstudies typically being performed prior to theRCTs. Such meta-confounding will make it difficultto attribute a systematic difference directly to theuse or non-use of a random allocation mechanism.Six of the eight reviews noted this problem andincorporated features in their evaluations toreduce the potential for metaconfounding.25–28,32,35 Two reviews made morestringent efforts to assess comparability than theothers. 25,26 Britton and colleagues restricted theselection of studies to be similar in terms ofintervention, setting, control therapy and outcomemeasure. MacLehose and colleagues assessed eachcomparison for the possibility of metaconfoundingand found that the most susceptible(i.e. those with differences in eligibility criteria andtime periods and no adjustment for severity ofdisease, comorbidity and other prognostic factors)had, on average, the largest discrepancies.Were the RCTs and non-randomised studiesshown to use similar study methodology in allrespects other than the allocation mechanism?Meta-confounding could also occur throughdifferences in other aspects of study design,beyond the use of randomisation. For example,the results of RCTs are known to vary according tothe quality of randomisation, especially theconcealment of allocation at recruitment. 23However, none of the reviews restricted theinclusion of RCTs to those with adequateconcealment or on any other methodologicalbasis. Only one review 26 assessed the comparabilityof randomised and non-randomised studies onany aspect of study quality (blinding).Discrepancies and similarities between studydesigns could be partly explained by differences inother unevaluated aspects of the methodologicalquality of the RCTs.Similarly, there will be differences in themethodological rigour of the non-randomisedstudies. Importantly, the possible biases of nonrandomisedstudies vary with study design. Onlyone review 27 restricted non-randomised studies tobe of a single design (historically controlledstudies). In all the others, RCTs were comparedwith non-randomised studies of a mixture ofdesigns.Were sensible, objective criteria used todetermine differences or equivalence of studyfindings?The manner in which results were judged to be‘equivalent’ or ‘discrepant’ varied widely betweenthe reviews and influenced the conclusions thatwere drawn. For example, Concato andcolleagues 33 deemed that the randomised andnon-randomised studies of mammographicscreening (comparison 34 in Table 3) hadremarkably similar results, whereas Ioannidis andcolleagues 34 classified them as discrepant. Only intwo reviews was the judgement made aggregatedacross the comparisons. 27,35 In all the otherreviews each individual topic was classified aseither equivalent or discrepant. Many of thecomparisons made at the level of a clinical topicwere based on very few data, for example, in theIoannidis review 34 on average for eachintervention five RCTs were compared with fournon-randomised studies. Hence the absence of astatistically significant difference cannot beinterpreted as evidence of ‘equivalency’ andclinically significant differences in treatmenteffects cannot be excluded. Conversely, thepresence of a statistically significant differencedoes not indicate that a clinically importantdifference does exist. Four reviews 26,28,34,35 moreusefully concentrated on describing the magnitudeof the differences, all four noting substantialdifferences occurring in some, but not all,comparisons. The Concato review 33 subjectivelyclassified all comparisons as being ‘remarkablysimilar’.The comparisons made in two reviews 33,34 of therelative variability of randomised and nonrandomisedresults can be considered flawedowing to the criteria used to compare variation.Concato and colleagues considered the range ofthe point estimates from observational studies andRCTs. 33 This comparison was confounded by thedifferent sample sizes used in observational andrandomised studies, and the number of studiesconsidered. On average, the RCTs in the Concatoreview were 25% smaller than the observationalstudies, hence greater variability in their results is19© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2: Health Technology Assessment 2003;
Page 3 and 4: Evaluating non-randomisedinterventi
Page 7: Health Technology Assessment 2003;
Page 23 and 24: © Queen’s Printer and Controller
Page 27 and 28: © Queen’s Printer and Controller
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 76 and 77: Empirical evaluation of the ability
Page 78 and 79: Empirical evaluation of the ability
Page 80 and 81:
Empirical evaluation of the ability
Page 82 and 83:
70TABLE 22 Comparison of concurrent
Page 84 and 85:
Page 86 and 87:
74TABLE 26 Comparison of methods of
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
80TABLE 33 Hypothetical example dem
Page 94 and 95:
82TABLE 34 Hypothetical example dem
Page 96 and 97:
Page 98 and 99:
Page 100 and 101:
Discussion and conclusions88histori
Page 102 and 103:
Discussion and conclusions90For exa
Page 104 and 105:
Discussion and conclusionsNon-rando
Page 107 and 108:
Health Technology Assessment 2003;
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121:
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

Create successful ePaper yourself

Delete template?

Save as template?