Evaluating non-randomised intervention studies - NIHR Health ...

More documents

Recommendations

Info

Appendix 4136DuRant, 1994 99The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredHow were subjects assigned to experimentalgroups?5.4 Any attempt to balance groups by designItem not covered9.2 Identification of prognostic factorsAs for 9.3 below9.3 Case-mix adjustmentWere appropriate variables or factorscontrolled for or blocked during the analysis?Were other potentially confounding variableshandled appropriately?This tool provides a list of 103 questions to aid theevaluation of research articles. The tool coversseveral study designs: experimental or quasiexperimental,survey designs and cross-sectionalstudies, retrospective chart reviews andretrospective studies and case–control studies.Some of the sections are general across all designs:‘introduction’ (8 items); ‘methods and procedures’(3 items); ‘results’ (12 items); and ‘discussion’(9 items). Additional sections relevant to nonrandomisedintervention studies are:‘experimental or quasi-experimental designs’(26 items) and ‘statistical analysis for experimentaldesigns’ (5 items). The tool took 15–25 minutes tocomplete. Again, it was not designed for use in asystematic review and although it covers themajority of issues relating to internal validity, thetool does not force the reader to answer thequestions in a systematic manner. It is also verylong and includes several irrelevant items. Thistool is really a critical appraisal tool designed toprompt thinking regarding quality and does notprovide a means of comparing the quality ofstudies included in a review.The DuRant tool was not judged to be suitable foruse in a systematic review.Fowkes and Fulton, 1991 107The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredItem not covered5.4 Any attempt to balance groups by designControl group acceptable? Matching orrandomisation9.2 Identification of prognostic factorsDistorting influences? Confounding factors9.3 Case-mix adjustmentDistorting influences? Distortion reduced byanalysisThis tool provides a list of six questions as achecklist for appraising a medical article. The toolaims to cover several study designs includingcross-sectional, cohort, controlled trials andcase–control studies. Each of the six questions listsitems to be scored ‘major problem’, minorproblem’ or ‘no problem’. The tool took around10 minutes to complete. This is another exampleof a checklist that was designed to promptthinking about quality but does not permit thesystematic assessment of quality across studies.Limited detail of what the items mean is providedin the actual checklists and the entire paper needsto be read to provide any guidance.The Fowkes tool was not judged to be suitable foruse in a systematic review.Hadorn and colleagues, 1996 102The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredFor cohort studies, the study groups were nottreated concurrently5.4 Any attempt to balance groups by designItem not covered9.2 Identification of prognostic factorsKnown prognostic factors for the outcome ofinterest or possible confounders were notmeasured at baseline9.3 Case-mix adjustmentA significant difference was found in one ormore baseline characteristics that are knownprognostic factors or confounders, but noadjustments were made for this in the analysisThis tool was developed for the rating of evidencefor clinical practice guidelines. It provides a list ofeight quality assessment criteria; each criterion listswhat the authors consider to be major and minorflaws. The criterion for allocation of patients totreatment groups has separate responses for RCTsand cohort or registry studies. The tool took 15–20minutes to complete. Although the tool covered anumber of validity issues, it was found to be fairlydifficult to use in its published format and perhapsconcentrated overly on pharmaceutical interventions.The Hadorn tool was not judged to be suitable foruse in a systematic review.
Health Technology Assessment 2003; Vol. 7: No. 27Spitzer and colleagues, 1990 105The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredSuitable assembly of comparison group5.4 Any attempt to balance groups by designKnown confounders accounted for by designAny methods to attempt comparabilitybetween groups, other than randomisation9.2 Identification of prognostic factorsSample size enables adequate precision insecondary variables reported (confoundingvariables or incidental findings)9.3 Case-mix adjustmentKnown confounders accounted for by analysisThis tool was developed for use in anepidemiological systematic review. It contains 20quality items and takes 10–20 minutes to complete.It was used as a quality assessment tool in one ofour sample of systematic reviews. 97 It scoresreasonably well on the required quality items andcan be applied to different types of study design;however, some of the questions are rather subjectiveand/or ambiguous (for example, ‘Selection biasaccounted for’) and some quality issues appear tobe covered by more than one item. Very littleguidance for completion of the tool is provided.Several response options are provided, but it isdifficult to differentiate these from each other (e.g.options include Uncertain/Incomplete/Substandard,Don’t know/Not reported, N/A and N/C).Overall, the Spitzer tool was not judged to besuitable for use in a systematic review.Vickers, 1995 110The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredItem not covered5.4 Any attempt to balance groups by designAdequacy of baseline matching9.2 Identification of prognostic factorsAll relevant variables included in baselinematching9.3 Case-mix adjustmentStratification/multivariate analysis adequateand appropriate?This checklist is a tool for critical appraisal,containing 21 items, several of which havefurther sub-questions. The tool takes up to20 minutes to complete. It covers a lot ofvalidity issues, but some of the wording israther ambiguous. This tool was designed toaid critical appraisal and prompts thinkingabout quality rather than providing a tool forsystematic quality assessment. It would bedifficult to use within the context of a systematicreview.The Vickers tool was not judged suitable for use ina systematic review.Weintraub, 1982 108The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredConcurrent or historical controls5.4 Any attempt to balance groups by designAssignment of treatment: randomised,matched or stratification or minimisation9.2 Identification of prognostic factorsComparability of treatment groups on specificcriteria9.3 Case-mix adjustmentItem not coveredThis tool was developed to evaluate criticallyreports of clinical drug trials and to aid in thewriting of protocols and papers. It contains 47items split into seven categories: population,treatments, experimental design details,compliance, data collection, control of bias anddata analysis. The checklist takes 10–15 minutes tocomplete and is aimed at studies of any designthat evaluate drug therapy. The checklist is notstructured in such a way that responses to thequestions are systematic and the items do notreally force the reader to make any qualityjudgements. There is a final section where anoverall assessment of study quality should bemade; however, it is not clear how thisassessment should follow from the precedingitems.The Weintraub tool is not a suitable qualityassessment tool for use in systematic reviews in itspresent format.137© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Page 1 and 2:
Health Technology Assessment 2003;
Page 3 and 4:
Evaluating non-randomisedinterventi
Page 5 and 6:
Page 7:
Page 11 and 12:
Page 13 and 14:
Page 15 and 16:
Page 17:
Page 21 and 22:
Page 23 and 24:
© Queen’s Printer and Controller
Page 25 and 26:
Page 27 and 28:
© Queen’s Printer and Controller
Page 29 and 30:
Page 31 and 32:
Page 33:
Page 36 and 37:
Evaluation of checklists and scales
Page 38 and 39:
Page 40 and 41:
Page 42 and 43:
Page 44 and 45:
32TABLE 8 Details of top 60 quality
Page 46 and 47:
Page 48 and 49:
Page 50 and 51:
38TABLE 10 Other domains: reporting
Page 52 and 53:
Page 54 and 55:
Page 56 and 57:
Use of quality assessment in system
Page 58 and 59:
Page 60 and 61:
Page 62 and 63:
Empirical estimates of bias associa
Page 64 and 65:
Page 66 and 67:
Page 68 and 69:
Page 70 and 71:
Page 72 and 73:
Page 74 and 75:
Page 76 and 77:
Empirical evaluation of the ability
Page 78 and 79:
Page 80 and 81:
Page 82 and 83:
70TABLE 22 Comparison of concurrent
Page 84 and 85:
Page 86 and 87:
74TABLE 26 Comparison of methods of
Page 88 and 89:
Page 90 and 91:
Page 92 and 93:
80TABLE 33 Hypothetical example dem
Page 94 and 95:
82TABLE 34 Hypothetical example dem
Page 96 and 97:
Page 98 and 99: Empirical evaluation of the ability
Page 100 and 101: Discussion and conclusions88histori
Page 102 and 103: Discussion and conclusions90For exa
Page 104 and 105: Discussion and conclusionsNon-rando
Page 107 and 108: Health Technology Assessment 2003;
Page 121: Health Technology Assessment 2003;
Page 124 and 125: Appendix 1data)) or (non-random$ or
Page 126 and 127: Appendix 2AuthorYearENDARESourcePub
Page 128 and 129: Appendix 2Author:Accession No:Endno
Page 130 and 131: Appendix 20 0 00Additional outcomes
Page 132 and 133: Appendix 2Endnote NoWas CMA conside
Page 134 and 135: 122AuthorOrigin aModified toolTool
Page 155 and 156: © Queen’s Printer and Controller
Page 185: Health Technology Assessment 2003;
Page 188 and 189: Health Technology Assessment Progra
Page 190: Health Technology Assessment Progra
show all

Evaluating non-randomised intervention studies - NIHR Health ...

Create successful ePaper yourself

Delete template?

Save as template?