Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

Evaluation of checklists and scales for assessing quality of non-randomised studiesprovoke thinking around quality issues but do notprovide a means of being systematic (e.g. tools tohelp critical appraisal of studies). Both of theseapproaches seem equally unacceptable.One of the main advantages of our review is thatwe identified a large number of quality assessmenttools, developed for use in a wide range of fieldsand published in a variety of formats. Although itwould not be possible to identify all availabletools, we did attempt to be as systematic aspossible. Our search covered a wide range ofdatabases and included both published andunpublished material. As a result, we are likely tohave identified a large proportion of availablequality assessment tools. In addition, all tools wereassessed in a systematic manner, by twoindependent reviewers.Nevertheless, several criticisms could be putforward. First, we could be criticised for beingover-inclusive as we did not require the tools tostate that their purpose was the methodologicalquality assessment of non-randomised studies. Werequired only that the tool had been or couldpotentially be used to assess such studies. Thisresulted in the inclusion of a wide range of tools,including some originally developed for RCTs. Wefeel that this approach can be justified as our aimwas to provide a comprehensive picture of hownon-randomised studies are being assessed andwhat other authors consider to be importantquality features. Furthermore, we had no a priorireason to assume that tools developed for RCTswould not be suitable for use with non-randomisedstudies, with or without adaptation.Second, the list of quality items against which weassessed the included tools was of our owngeneration, and as such, could be questioned. Ourstrategy was to take everything that is known aboutquality for randomised studies and add what weknow to be different about non-randomisedstudies. It should be noted that the aim of thetaxonomy was to provide an aid to data extractionof the identified tools as opposed to providing anexhaustive or comprehensive list of quality items.The alternative would have been to extract detailsof all the items from each quality assessment tooland then try to group them, but the sheer volumeof tools, and the use of varying terminology andquestion phrasing, would have made thisimpossible.Furthermore, the selection process that we used toidentify the ‘best’ tools (i.e. items in five out of sixinternal validity domains plus at least three of fourpre-specified core items) meant that those in thefinal sample of 14 would not necessarily have thebest coverage of the six domains. However, theseselection criteria were chosen first to reflect oura priori position that degree of selection bias is afundamental issue that needs to be addressed byquality assessment tools (hence the selection of thefour core items) and second because a morerestrictive policy, for example requiring items inall six domains, would have led to the finalassessment of only six tools for their ‘usability’ insystematic reviews.Another criticism that could be made relates toour assessment of the ‘usability’ of the best tools.This was based on their application to only threenon-randomised studies and could benefit fromrepetition on a larger sample of studies.Future research should include:●●●further appraisal of the quality criteria that weselectedconsideration of the creation of a new tool orthe revision of an existing onemore detailed examination of tool ‘usability’,for example, tools may be more or less usefulaccording to field or type of intervention.42
Health Technology Assessment 2003; Vol. 7: No. 27Chapter 5Use of quality assessment in systematic reviewsof non-randomised studiesIntroductionAs we have discussed in some detail in previouschapters, the principal difference betweenrandomised and non-randomised studies lies inthe latter’s considerable susceptibility to selectionbias. Concealed randomisation specificallyremoves the possibility of selection bias orconfounding in RCTs, i.e. any differences betweenthe groups are attributable to chance or to theintervention, all else being equal. 20 For nonrandomisedstudies, confounding between groupsis likely. Whatever the method of allocation, thereasons for the choice of an intervention can beinfluenced by subtle clues that are not easilyidentifiable but which may relate, for example, tothe patient’s prognosis. 3 Treatment selection maybe confounded by differences in case-mix, but alsoby use of concomitant healthcare interventions orvariation in the outcome assessment process, forexample where allocation is influenced bygeographical location or temporal differences. Anyassessment of the degree of selection biasintroduced by non-random methods of allocationis of primary importance for systematic reviews ofnon-randomised studies.Empirical evidence for the impact of assessingstudy quality on the results of systematic reviews islimited. However, work on RCTs generallyindicates that low-quality trials can lead to adistortion of true treatment effects. 20,21,59,115,116Furthermore, it has been shown that the choice ofquality scale can dramatically influence theinterpretation of meta-analyses, and can evenreverse conclusions regarding the effectiveness ofan intervention. 59 This lends support to the needfor careful choice of a quality assessment tool andfor consideration of the impact of study quality ona review’s conclusions.Recent guidelines for the reporting of meta-analysesof observational studies 117 recommend the“assessment of confounding, study quality andheterogeneity” to be clearly reported in themethods section of these reviews but moreoverthat “thorough specification of quality assessmentcan contribute to understanding some of thevariations in the observational studies themselves”.The results of quality assessment can be used in asystematic review in several ways, 92 includingforming inclusion criteria for the review orprimary analysis; informing a sensitivity analysisor meta-regression; weighting studies; orhighlighting areas of methodological qualitypoorly addressed by the included studies and theimpact of this on the review’s conclusions.The objectives of the study described in thischapter are to estimate the extent to which qualityassessment has been used in systematic reviews ofnon-randomised studies and can adequatelyidentify the sources of potential bias, and todescribe the methods used to incorporate theresults of the quality assessment into theconclusions of a review.MethodsStudy selectionWe sought systematic reviews evaluating theintended effect of an intervention and thatincluded at least one non-randomised study.Data sourcesThe NHS Centre for Reviews and DisseminationDatabase of Abstracts of Reviews of Effectiveness(DARE) was used to identify reviews. This is adatabase of quality-assessed systematic reviewsidentified by handsearching key major medicaljournals, regular searching of electronicbibliographic databases and scanning ‘grey’literature since 1994 (further details aboutDARE can be found athttp://agatha.york.ac.uk/darehp.htm). All reviewsentered in DARE up to December 1999 werescreened for inclusion. Further searches ofprimary databases with the aim of identifyingadditional systematic reviews not indexed inDARE were not carried out; however, any suchreviews identified from the other searches for theproject (see Chapters 3 and 4) were assessed forinclusion.43© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2:
Health Technology Assessment 2003;
Page 3 and 4: Evaluating non-randomisedinterventi
Page 5 and 6: Health Technology Assessment 2003;
Page 7: Health Technology Assessment 2003;
Page 23 and 24: © Queen’s Printer and Controller
Page 27 and 28: © Queen’s Printer and Controller
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 76 and 77: Empirical evaluation of the ability
Page 82 and 83: 70TABLE 22 Comparison of concurrent
Page 86 and 87: 74TABLE 26 Comparison of methods of
Page 92 and 93: 80TABLE 33 Hypothetical example dem
Page 94 and 95: 82TABLE 34 Hypothetical example dem
Page 100 and 101: Discussion and conclusions88histori
Page 102 and 103: Discussion and conclusions90For exa
Page 104 and 105:
Discussion and conclusionsNon-rando
Page 107 and 108:
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121:
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

Create successful ePaper yourself

Delete template?

Save as template?