38TABLE 10 Other domains: reporting and external validityNewcastle-Bracken 104 CASP 64 Cowley 109 Downs 85 DuRant 99 Fowkes 107 Hadorn 102 Ottawa 66 Reisch 111 Spitzer 105 Thomas 65 Vickers 110 Weintraub 108 Zaza 86#1 Background1.1. Background information ✓ ✓1.1. provided#2 Sample2.1. Retrospective/prospective ✓ ✓2.2. Inclusion/exclusion criteria ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓2.3. Sample size ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓2.4. Representative ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓#3 Interventions3.1. Clear specification of ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓3.1. <strong>intervention</strong>s#4 Outcomes4.1. Clear specification of ✓ ✓ ✓ ✓ ✓4.1. outcomes#11 Interpretation11.1. Appropriately based on ✓ ✓ ✓ ✓11.1. results11.2. Assessed strength of ✓ ✓ ✓11.1. evidence11.3. Application/implications ✓ ✓ ✓ ✓#12 Presentation12.1. Completeness, clarity, ✓ ✓ ✓ ✓11.1. structureEvaluation of checklists and scales for assessing quality of <strong>non</strong>-<strong>randomised</strong> <strong>studies</strong>
<strong>Health</strong> Technology Assessment 2003; Vol. 7: No. 27the tools in this group were specifically developedfor use in a systematic review, 65,66,85,86,109 theirmain advantage being that the questions andresponses were phrased in such a way as to forcethe reviewers to be systematic in their assessmentof individual <strong>studies</strong>, and ensure that betweenstudyquality can be compared.The length and complexity of the tools varied, butmost provided guides for completion of the toolsand three of the tools were considered ‘easy to use’in our assessment. 65,85,109 Although a certaindegree of judgement is required for completion ofany tool (including phrases such as‘appropriateness of allocation’, ‘trulyrepresentative’ and ‘adequately described’), theguides help the reader to interpret the questionsand response options. In several of thesetools, 66,85,109,111 the responses to individualquestions could be used to contribute to an overalljudgement of study quality, using either number ofitems fulfilled 66,109,111 or summary scores. 85 Two ofthe selected tools 65,86 used a ‘mixed-criteria’approach, including specific factual questionsabout the study design and also requiring moregeneral judgements as to the degree of biaspresent in a study.A summary of each of the six selected toolsfollows, with details of how they covered the fourcore items provided in Appendix 5.CowleyCowley 109 developed a quality assessment tool foruse in a systematic review of total hip replacement.Separate items were provided for RCTs,comparative <strong>studies</strong> and uncontrolled case series.The tool provides a list of 13 questions for thequality assessment of comparative <strong>studies</strong>, splitinto two sections: ‘key criteria’ (seven items) and‘other criteria’ (six items). No details of themethod of item generation were provided. Studieswere rated A, B or C, according to the number of‘key’ and ‘other’ criteria that were met. Key criteriaincluded the method of assignment of patients togroups, matching of groups for key prognosticvariables, appropriate statistical analysis andspecification of <strong>intervention</strong>s and outcomes.Several items were topic-specific. No items forsample size and method of sample selection wereprovided. A mixture of items relating to bothreporting and methodological quality wereincluded. The authors did not attempt to establishthe validity or reliability of the tool, and did notprovide a guide to completion. However, we foundthe criteria quick and easy to apply, taking 5–15minutes to complete the checklist. With somemodifications, the Cowley tool was judged to besuitable for use in a systematic review.DownsDowns and Black 85 developed a scale to assess themethodological quality of <strong>randomised</strong> and <strong>non</strong><strong>randomised</strong><strong>studies</strong>. The tool was used in twosystematic reviews in our sample, 26,112 and appearsto have been developed for such use. It provides alist of 27 questions to measure study quality, splitinto four sections: ‘reporting’ (10 items); ‘externalvalidity’ (three items); ‘internal validity – bias’(seven items); and ‘internal validity – confounding(selection bias)’ (seven items). Epidemiologicalprinciples and methodological literature were usedto generate an initial version of the tool, whichwas then piloted by experienced epidemiologistsand statisticians and a revised version produced.The tool was found to be fairly comprehensive;however, questions regarding the allocationmechanism used relate only to <strong>randomised</strong><strong>studies</strong>, and no items relating to baselinecomparability were included. Nevertheless, it wasfound to be easy to use, with yes/no/can’t tellanswers and clear descriptions of how to scoreitems. The tool authors found the validity andreliability of the tool to be reasonably high, exceptfor the three items on external validity. Althoughrelatively quick to complete (10–20 minutes), thetool is fairly long and a large number of questionsrelate to reporting as opposed to validity. TheDowns tool was judged to be suitable for use in asystematic review, although some modification,such as the addition of the missing items (e.g.baseline comparability), may be warranted.Newcastle–OttawaThe Newcastle–Ottawa tool 66 was designed for usein an epidemiological systematic review, and canbe used as either a scale or a checklist. It wasdeveloped using a Delphi process to definevariables for data extraction, was then tested onsystematic reviews and further refined. Separatetools were developed for cohort and case–control<strong>studies</strong>, although only the tool for cohort <strong>studies</strong> isconsidered here. The tool contains eight items,categorised into three groups: selection,comparability and outcome. For each item a seriesof response options are provided. For example, forthe consideration of the representativeness of theexposed cohort, response options include trulyrepresentative, somewhat representative, selectedgroup or no description of exposed cohort. A starsystem is used to allow a visual semi-quantitativeassessment of study quality, such that the highestquality <strong>studies</strong> are awarded a maximum of one39© Queen’s Printer and Controller of HMSO 2003. All rights reserved.
Change language