Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

Evaluation of checklists and scales for assessing quality of non-randomised studiesTABLE 7 Selected features of identified quality assessment toolsTool feature Best tools Top tools Other tools Total(n = 14) (n = 46) (n = 133) (n = 193)Publication statusPublished 11 (79%) 46 (100%) 133 (100%) 190 (98%)Unpublished 3 (21%) 0 0 3 (2%)Type of toolChecklist 11 (79%) 18 (39%) 75 (56%) 104 (54%)Scale 3 (21%) 28 (61%) 58 (44%) 89 (46%)If scaleEqual weighting 1 (33%) 14 (50%) 30 (51%) 45 (50%)Unequal weighting 2 (67%) 14 (50%) 16 (29%) 32 (37%)Not described 0 0 12 (20%) 12 (13%)Tool originNew tool 12 (86%) 29 (63%) 101 (76%) 142 (74%)Modified tool 2 (14%) 17 (37%) 32 (24%) 51 (26%)Tool developmentLiterature/consensus 6 (43%) 26 (57%) 41 (31%) 73 (37%)Survey/expert panel 3 (21%) 3 (7%) 5 (4%) 11 (5%)Not described 8 (36%) 17 (36%) 87 (65%) 112 (58%)Validity/reliabilityValidity 1 (7%) 3 (7%) 5 (4%) 9 (5%)Reliability 2 (14%) 22 (48%) 39 (29%) 54 (28%)Tool purposeStudy planning 0 2 (4%) 1 (1%) 3 (2%)Statistical analysis 0 2 (4%) 13 (10%) 15 (8%)Critical appraisal 5 (36%) 12 (26%) 22 (16%) 39 (20%)Guidelines 0 1 (2%) 0 1 (1%)Peer review 0 0 1 (1%) 1 (1%)Reporting 1 (7%) 1 (2%) 3 (2%) 5 (3%)Systematic review 8 (57%) 28 (61%) 93 (70%) 129 (65%)Number of itemsMedian (range) 22 (8–103) 17 (7–49) 10 (3–162) 12 (3–162)30description of the interventions. The basis for theweighting system adopted was rarely described.The method used to develop the tool was rarelyreported in detail. Most tools were based on someform of consensus, gleaned either from themethodological literature, existing qualityassessment tools or expert opinion. A total of 11tools used either surveys or panels of experts toselect the quality items and piloted and/orcirculated the list of items to experts for commentand revision. 46,66,83–91 Twenty tools gave noindication of how the items listed had beengenerated.A very small minority of tools attempted toestablish the validity of the tool (only four of thetop 60 tools). Of these, two found face and contentvalidity to be reasonable. 84,85 All four stated thatthey had examined criterion validity. Two of thesecompared their tools with that of Detsky andcolleagues (a reduced version of the Chalmersscale for assessing the quality of RCTs): 92 one 84found a significant difference in mean scores( p < 0.001), but a similar rank order of scores(Kendall’s W = 0.74); the other 93 did not presentthe results of the comparison. Downs and Black 85found a high correlation (Spearman correlationcoefficient 0.90) between the total score from theirtool (for both randomised and non-randomisedstudies) and that obtained for the Standards ofReporting Trials Group tool 94 (used onrandomised studies only). Linde and colleagues 95found similar results from their tool and that ofJadad and colleagues, 46 but did not carry out astatistical comparison.Twenty-four of the top 60 tools assessed inter-raterreliability. Two of these stated only that agreementwas mediocre or good and nine provided onlypercentage agreement between raters (range70–94%). Where provided (13 studies), kappa orother correlation coefficients were generally >0.75(n = 12), indicating good agreement. A few
© Queen’s Printer and Controller of HMSO 2003. All rights reserved.TABLE 8 Details of top 60 quality assessment tools (14 ‘best’ tools denoted by shaded areas)Author a Originality b Type of No. of NRS Used in Tool Tool Tool 5 IV Core 5 domains andtool c items d items? e reviews? f purpose g validity h reliability i domains j items k 3 core items lAntczak, 1986 76 m; Chalmers, 1981 51 s u 30 n 2 7 na na Y 0Audet, 1993 176 m; Poynard, 1988 403 s e 17 n 1 7 na IRR Y 1Bours, 1998 73 m; van der Windt, 1995 71 s u 18 y n 7 na IaRR Y 1Bracken, 1989 104 n c 36 n n 6 na na Y 4 YCameron, 2000 83 n c 36 n n 7 na na Y 0Campos-Outcalt, 1995 177 n s e 7 n n 7 na na Y 1Carter, 1994 178 m; Chalmers, 1981 51 s u 12 n n 7 na na Y 0CASP, 1999 64 n c 10 n n 3 na na Y 3 YChalmers, 1981 51 n s u 36 n 15 3 na na Y 0Cho, 1994 84 m; Spitzer, 1990 105 s e 31 n n 3 FC, Cr IRR Y 2Cochrane MIG, 2002 179 m; Chalmers, 1981 51 s e 12 n CC 7 na na Y 1Coleridge Smith, 1999 97 n c 49 y n 7 na IRR Y 2Cowley, 1995 109 n c 17 n n 7 na na Y 3 YCuddy, 1983 180 n c 17 n n 3 na na Y 2Dawson-Saunders, 1990 98 n c 49 y n 3 na na Y 2de Oliveira, 1995 96 m; Chalmers, 1981 51 s u 30 n 1 7 na IRR Y 1de Vet, 1997 72 m; ter Riet, 1990 68 s e 15 n n 7 na IRR 67 Y 2Downs, 1998 85 n s u 27 n 3 3 FC, Cr IRR; T-R Y 3 YDuRant, 1994 99 n c 103 y n 3 na na Y 3 YFowkes, 1991 107 n c 23 n 1 3 na na Y 3 YFriedenreich, 1993 100 n c 29 y n 7 na na Y 2Gardner, 1986 181 n c 26 n n 2 na na Y 2Glantz, 1997 182 m; Chalmers, 1981 51 s u 20 n n 7 na na Y 0Gordis, 1990 101 n c 17 y 1 3 na na Y 2Greenhalgh, 1997 183 n c 11 n n 3 na na Y 1Gurman, 1978 184 n s u 14 n n 2 na na Y 1Guyatt, 1993 79 n c 12 n 1 3 na IRR 185 Y 0Hadorn, 1996 102 m; US Task Force, 1989 378 c 41 y n 4 na na Y 3 YKay, 1996 186 n s e 20 n n 7 na na Y 0Koes, 1991 70 m; ter Riet, 1990 68 s u 17 n 3 7 na IaRR Y 0Kreulen, 1998 187 m; Antzcak, 1986 76 s u 17 n n 7 na IRR Y 1Kwakkel, 1997 188 n s e 16 n n 7 na IRR Y 1Lee, 1997 189 m; Cook, 1979 50 s e 7 n n 7 na IaRR Y 2Levine, 1980 190 n s e 30 n 1 3 na na Y 1Linde, 1999 95 n s e 7 n 1 2/7 Cr na Y 0MacMillan, 1994 191 m; Chalmers, 1981 51 s u 9 n n 7 na IaRR Y 1Maziak, 1998 192 n s e 10 n n 7 na IRR Y 0Meijman, 1995 193 m; Fowkes, 1991 107 s e 11 n n 2 na no stats Y 1continuedHealth Technology Assessment 2003; Vol. 7: No. 2731
Page 1 and 2: Health Technology Assessment 2003;
Page 3 and 4: Evaluating non-randomisedinterventi
Page 7: Health Technology Assessment 2003;
Page 23 and 24: © Queen’s Printer and Controller
Page 27 and 28: © Queen’s Printer and Controller
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 56 and 57: Use of quality assessment in system
Page 62 and 63: Empirical estimates of bias associa
Page 76 and 77: Empirical evaluation of the ability
Page 82 and 83: 70TABLE 22 Comparison of concurrent
Page 86 and 87: 74TABLE 26 Comparison of methods of
Page 92 and 93:
80TABLE 33 Hypothetical example dem
Page 94 and 95:
82TABLE 34 Hypothetical example dem
Page 96 and 97:
Empirical evaluation of the ability
Page 98 and 99:
Empirical evaluation of the ability
Page 100 and 101:
Discussion and conclusions88histori
Page 102 and 103:
Discussion and conclusions90For exa
Page 104 and 105:
Discussion and conclusionsNon-rando
Page 107 and 108:
Health Technology Assessment 2003;
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121:
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?