Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

32TABLE 8 Details of top 60 quality assessment tools (14 ‘best’ tools denoted by shaded areas) (cont’d)Author a Originality b Type of No. of NRS Used in Tool Tool Tool 5 IV Core 5 domains andtool c items d items? e reviews? f purpose g validity h reliability i domains j items k 3 core items lMelchart, 1994 93 n s e 15 n n 7 Cr IaRR Y 0Miller, 1995 87 n s u 12 y 1 7 na IaRR Y 2Moncrieff, 1998 195 n s e 30 n n 3 na IRR; IC Y 1Morley 1996 196 m; Chalmers, 1981 51 s u 30 n 1 2/6 na IRR Y 2Mulrow, 1986 197 n c 13 n n 7 na IaRR Y 2Newcastle–Ottawa 66 n s u 8 y n 7 na na Y 3 YNicolucci, 1989 77 m; Chalmers, 1981 51 s u 16 n 1 7 na IaRR Y 0Reisch, 1989 111 n s e 57 n CC 1/3 na IRR 114 Y 3 YSalisbury, 1997 198 n c 45 n n 7 na na Y 2Schechter, 1991 199 n c 30 n n 3 na na Y 2Sheldon, 1993 200 n c 36 n n 3 na na Y 2Spitzer, 1990 105 n c 20 n 1 7 na na Y 4 YTalley, 1993 106 n c 29 n n 7 na na Y 0Thomas 65 n c 21 n n 7 na na Y 3 Yvan der Windt, 1995 71 m; Koes, 1991 70 s u 17 n 1 7 na IaRR Y 1Vickers, 1996 201 n c 12 n n 7 na no stats Y 1Vickers, 1995 110 n c 21 n n 3 na na Y 3 YWeintraub, 1982 108 m; Lionel, 1970 315 c 47 n n 1/2/3 na na Y 3 YWilson, 1992 103 n c 10 y n 7 na na Y 1Wingood, 1996 202 n c 16 n n 7 na na Y 0Wright, 1995 203 n c 13? n n 7 na na Y 0Zaza, 2000 86 n c 22 n n 7 na na Y 3 Ya Name of tool or principal author.b n, New tool; m, modification of existing tool.c c, Checklist; s, scale, u, unequal weighting scheme, e, equal weighting scheme.d Total number of items.e Did the tool include items specific to non-randomised studies? n, no; y, yes.fHas the tool been used in identified sample of systematic reviews?: n, no; CC, used in Cochrane reviews; or give number of reviews.g What purpose was the tool designed for? 1, list for planning a study; 2, list for assessing a statistical/methodological analysis; 3, list for evaluating a study when considering thepractical application of the intervention; 4, list for evaluating/grading study recommendations when producing recommendations/guidelines; 5, list for peer reviewing; 6, list forreporting a study; 7, list for assessing studies included in a systematic review.h Was an attempt made to establish tool validity? na, not assessed; FC, face and content; Cr, criterion; Co, content.iWas an attempt made to establish tool reliability? na, not assessed; IRR, inter-rater reliability; IaRR, intra-rater reliability; IC, internal consistency; T-R, test-retest; a, % agreementassessed.jWere at least 5 internal validity domains covered? Y, yes.k Number of core items covered.lWere at least 5 internal validity domains and at least 3 of the 4 key items covered? Y, yes.Evaluation of checklists and scales for assessing quality of non-randomised studies
Health Technology Assessment 2003; Vol. 7: No. 27Potential toolsn = 213Included toolsn = 193Excluded:levels of evidence: n = 7not quality assessment tools: n = 13‘top tools’ an = 60Excluded from contentanalysis: n = 11‘Best tools’ bn = 14Suitable for use in asystematic reviewn = 6a ≥5 internal validity domains metb ≥3 core items includedFIGURE 2 Flowchart of tool selection processstudies found the level of agreement to bedependent on the experience of the raters.de Oliveira and colleagues 96 achieved higheragreement for experienced raters, whereasColeridge Smith 97 found higher agreement fortwo methodologists combined rather thanclinicians alone or for a clinical/methodologistpairing. Other aspects of reliability were rarelyexamined.The majority of the identified tools were designedfor use in a systematic review (and most were infact published within the context of a systematicreview). Of the remainder, most were criticalappraisal tools, designed to assist the reader whenconsidering the usefulness of a particularintervention in their situation, rather than to beused to compare quality across studies.The number of items in the tools ranged from 3 to162, with a median of 12 items. Those toolsselected as ‘top tools’ or ‘best tools’ had a highermedian number of items compared with theunselected tools. Those with the largest number ofitems (>50) were more likely to have separatesections and questions according to study design(e.g. including items specific to RCTs, cohortstudies, cross-sectional studies).Possibly the most relevant aspect of these tools forapplication in practice is how well they coverquestions of internal validity and, in particular,whether they covered the four core items. Elevenof the 193 tools did not specify the items includedin the tools. Of the remainder, 60 tools covered atleast five of the six internal validity domains.Individual star plots demonstrating how well, or towhat extent, these tools covered each of the sixinternal validity domains are provided in Figure 3.These plots indicate a significant variation in howwell these domains were covered. Figures 4 and 5provide information on how often each domainwas missed and how well the tools covered eachdomain. The domain most commonly missed wassoundness of information (38%), with only 37% oftools including an item relating to whether theinterventions had been received by the studyparticipants and only 25% including an item onwhether or not the outcomes were attributable.All of the top 60 tools included one of the two33© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2: Health Technology Assessment 2003;
Page 3 and 4: Evaluating non-randomisedinterventi
Page 7: Health Technology Assessment 2003;
Page 23 and 24: © Queen’s Printer and Controller
Page 27 and 28: © Queen’s Printer and Controller
Page 36 and 37: Evaluation of checklists and scales
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 76 and 77: Empirical evaluation of the ability
Page 82 and 83: 70TABLE 22 Comparison of concurrent
Page 86 and 87: 74TABLE 26 Comparison of methods of
Page 92 and 93: 80TABLE 33 Hypothetical example dem
Page 94 and 95:
82TABLE 34 Hypothetical example dem
Page 96 and 97:
Empirical evaluation of the ability
Page 98 and 99:
Empirical evaluation of the ability
Page 100 and 101:
Discussion and conclusions88histori
Page 102 and 103:
Discussion and conclusions90For exa
Page 104 and 105:
Discussion and conclusionsNon-rando
Page 107 and 108:
Health Technology Assessment 2003;
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121:
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

Create successful ePaper yourself

Delete template?

Save as template?