Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

NHS R&D HTA ProgrammeThe NHS R&D Health Technology Assessment (HTA) Programme was set up in 1993 to ensurethat high-quality research information on the costs, effectiveness and broader impact of healthtechnologies is produced in the most efficient way for those who use, manage and provide carein the NHS.Initially, six HTA panels (pharmaceuticals, acute sector, primary and community care, diagnosticsand imaging, population screening, methodology) helped to set the research priorities for the HTAProgramme. However, during the past few years there have been a number of changes in and aroundNHS R&D, such as the establishment of the National Institute for Clinical Excellence (NICE) andthe creation of three new research programmes: Service Delivery and Organisation (SDO); New andEmerging Applications of Technology (NEAT); and the Methodology Programme.The research reported in this monograph was identified as a priority by the HTA Programme’sMethodology Panel and was funded as project number 96/26/99.The views expressed in this publication are those of the authors and not necessarily those of theMethodology Programme, HTA Programme or the Department of Health. The editors wish toemphasise that funding and publication of this research by the NHS should not be taken as implicitsupport for any recommendations made by the authors.Criteria for inclusion in the HTA monograph seriesReports are published in the HTA monograph series if (1) they have resulted from workcommissioned for the HTA Programme, and (2) they are of a sufficiently high scientific qualityas assessed by the referees and editors.Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search,appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permitthe replication of the review by others.Methodology Programme Director: Professor Richard LilfordHTA Programme Director: Professor Kent WoodsSeries Editors:Professor Andrew Stevens, Dr Ken Stein, Professor John Gabbay,Dr Ruairidh Milne and Dr Rob RiemsmaManaging Editors:Sally Bailey and Sarah Llewellyn LloydThe editors and publisher have tried to ensure the accuracy of this report but do not accept liabilityfor damages or losses arising from material published in this report. They would like to thank thereferees for their constructive comments on the draft document.ISSN 1366-5278© Queen’s Printer and Controller of HMSO 2003This monograph may be freely reproduced for the purposes of private research and study and may be included in professional journalsprovided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising.Applications for commercial reproduction should be addressed to HMSO,The Copyright Unit, St Clements House,2–16 Colegate, Norwich, NR3 1BQ.Published by Gray Publishing, Tunbridge Wells, Kent, on behalf of NCCHTA.Printed on acid-free paper in the UK by St Edmundsbury Press Ltd, Bury St Edmunds, Suffolk.M
Health Technology Assessment 2003; Vol. 7: No. 27AbstractEvaluating non-randomised intervention studiesJJ Deeks, 1* J Dinnes, 2 R D’Amico, 1 AJ Sowden, 3 C Sakarovitch, 1 F Song, 4M Petticrew 5 and DG Altman 1In collaboration with the International Stroke Trial and the European Carotid Surgery Trial Collaborative Groups1 Centre for Statistics in Medicine, Institute of Health Sciences, Oxford, UK2 Southampton Health Technology Assessments Centre, University of Southampton, UK3 NHS Centre for Reviews and Dissemination, University of York, UK4 Department of Public Health and Epidemiology, University of Birmingham, UK5 MRC Social and Public Health Sciences Unit, University of Glasgow, UK* Corresponding authorObjectives: To consider methods and related evidencefor evaluating bias in non-randomised interventionstudies.Data sources: Systematic reviews and methodologicalpapers were identified from a search of electronicdatabases; handsearches of key medical journals andcontact with experts working in the field. Newempirical studies were conducted using data from twolarge randomised clinical trials.Methods: Three systematic reviews and new empiricalinvestigations were conducted. The reviews considered,in regard to non-randomised studies, (1) the existingevidence of bias, (2) the content of quality assessmenttools, (3) the ways that study quality has been assessedand addressed. (4) The empirical investigations wereconducted generating non-randomised studies fromtwo large, multicentre randomised controlled trials(RCTs) and selectively resampling trial participantsaccording to allocated treatment, centre andperiod.Results: In the systematic reviews, eight studiescompared results of randomised and non-randomisedstudies across multiple interventions using metaepidemiologicaltechniques. A total of 194 tools wereidentified that could be or had been used to assessnon-randomised studies. Sixty tools covered at leastfive of six pre-specified internal validity domains.Fourteen tools covered three of four core items ofparticular importance for non-randomised studies. Sixtools were thought suitable for use in systematicreviews. Of 511 systematic reviews that included nonrandomisedstudies, only 169 (33%) assessed studyquality. Sixty-nine reviews investigated the impact ofquality on study results in a quantitative manner. Thenew empirical studies estimated the bias associatedwith non-random allocation and found that the biascould lead to consistent over- or underestimations oftreatment effects, also the bias increased variation inresults for both historical and concurrent controls,owing to haphazard differences in case-mix betweengroups. The biases were large enough to lead studiesfalsely to conclude significant findings of benefit orharm. Four strategies for case-mix adjustment wereevaluated: none adequately adjusted for bias inhistorically and concurrently controlled studies. Logisticregression on average increased bias. Propensity scoremethods performed better, but were not satisfactory inmost situations. Detailed investigation revealed thatadequate adjustment can only be achieved in theunrealistic situation when selection depends on a singlefactor.Conclusions: Results of non-randomised studiessometimes, but not always, differ from results ofrandomised studies of the same intervention. Nonrandomisedstudies may still give seriously misleadingresults when treated and control groups appear similarin key prognostic factors. Standard methods of case-mixadjustment do not guarantee removal of bias. Residualconfounding may be high even when good prognosticdata are available, and in some situations adjustedresults may appear more biased than unadjusted results.Although many quality assessment tools exist and havebeen used for appraising non-randomised studies, mostomit key quality domains. Healthcare policies basedupon non-randomised studies or systematic reviews ofnon-randomised studies may need re-evaluation if theuncertainty in the true evidence base was not fullyappreciated when policies were made. The inability ofiii© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2: Health Technology Assessment 2003;
Page 3: Evaluating non-randomisedinterventi
Page 7: Health Technology Assessment 2003;
Page 23 and 24: © Queen’s Printer and Controller
Page 27 and 28: © Queen’s Printer and Controller
Page 36 and 37: Evaluation of checklists and scales
Page 44 and 45: 32TABLE 8 Details of top 60 quality
Page 50 and 51: 38TABLE 10 Other domains: reporting
Page 54 and 55:
Evaluation of checklists and scales
Page 56 and 57:
Use of quality assessment in system
Page 58 and 59:
Page 60 and 61:
Page 62 and 63:
Empirical estimates of bias associa
Page 64 and 65:
Page 66 and 67:
Page 68 and 69:
Page 70 and 71:
Page 72 and 73:
Page 74 and 75:
Page 76 and 77:
Empirical evaluation of the ability
Page 78 and 79:
Page 80 and 81:
Page 82 and 83:
70TABLE 22 Comparison of concurrent
Page 84 and 85:
Page 86 and 87:
74TABLE 26 Comparison of methods of
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
80TABLE 33 Hypothetical example dem
Page 94 and 95:
82TABLE 34 Hypothetical example dem
Page 96 and 97:
Page 98 and 99:
Page 100 and 101:
Discussion and conclusions88histori
Page 102 and 103:
Discussion and conclusions90For exa
Page 104 and 105:
Discussion and conclusionsNon-rando
Page 107 and 108:
Health Technology Assessment 2003;
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121:
Page 124 and 125:
Appendix 1data)) or (non-random$ or
Page 126 and 127:
Appendix 2AuthorYearENDARESourcePub
Page 128 and 129:
Appendix 2Author:Accession No:Endno
Page 130 and 131:
Appendix 20 0 00Additional outcomes
Page 132 and 133:
Appendix 2Endnote NoWas CMA conside
Page 134 and 135:
122AuthorOrigin aModified toolTool
Page 136 and 137:
Page 138 and 139:
Page 140 and 141:
Page 142 and 143:
Page 144 and 145:
Page 146 and 147:
Page 148 and 149:
Appendix 4136DuRant, 1994 99The typ
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
© Queen’s Printer and Controller
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
Page 163 and 164:
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185:
Page 188 and 189:
Health Technology Assessment Progra
Page 190:
Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

Create successful ePaper yourself

Delete template?

Save as template?