Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 11 Breakdown of reviews including non-randomised studiesStudy designs included No. of reviews No. using quality Proportion using qualityassessment assessment (%)NRS only 32 5 16NRS + RCTs 216 99 46NRS + RCTs + uncontrolled 116 34 29NRS + uncontrolled 41 9 22Not clearly reported 106 21 20Total 511 168 33NRS, non-randomised study.least one RCT and in 87 reviews a term such as‘controlled trial’ was used to describe the studiesso that RCTs were not clearly distinguished fromnon-randomised studies (Table 11). Those reviewswhich included both RCTs and non-randomisedstudies were more likely to have conducted qualityassessment (conducted in 46% of cases) than thosewith non-randomised studies only (16%) or thosealso including uncontrolled studies. Full details ofthe review methods are provided in Appendix 6and review results in Appendix 7.Source of assessment toolsOf the 168 reviews claiming to have carried outquality assessment, 69 (41%) stated they had usedexisting quality assessment tools, 28 (17%) mademodifications to existing tools and 71 (42%)developed their own tool (Figure 6).Closer inspection of the papers cited in the 69reviews that used existing tools revealed that 10 ofthe tools (cited in 13 of the reviews) were not infact quality assessment tools, but weremethodological papers, textbooks or manuals onhow to conduct reviews 118–123 or design primarystudies. 50,124,125 Nineteen reviews used a variety ofversions of the ‘levels of evidence’ frameworkoriginally developed by Sackett 126 and did notassess quality beyond identifying the study design.Amongst the remaining 37 reviews, the mostfrequently used tools were originally developed forthe assessment of RCTs (Figure 6): the Chalmersscale, 51 or modifications of it 76,77,92,96 (six reviews);the Maastricht Criteria List 69,70 (three reviews);and the Jadad scale 46 (three reviews). Othercommonly used tools were ones developed for usein public health 127 (used in three reviews) and atool developed by Oakley and colleagues 128developed for and used by the same authors forreviews of the health promotion literature (threereviews).Of the 28 reviews that modified existing tools,only one 26 reported details about how the tool(originally published by Downs and Black 85 ) hadbeen modified and also attempted to examine thevalidity of the tool. Twelve of these reviewsreported inter-rater reliability, all producing kappacoefficients of >0.70 or percentage agreement ofat least 80%.In the review by MacLehose and colleagues, 26 theitems considered to be the most important andunambiguous ‘quality items’ were identified bythree authors and were categorised into one offour dimensions: quality of reporting, internalvalidity, susceptibility to confounding and externalvalidity. The internal consistency of the individualitems contributing to the first three of thesedimensions was described using Cronbach’s α.The resulting low α values indicated thatindividual items were poorly correlated with thesum of scores for the items, suggesting that theitems were not assessing a homogeneous aspect ofquality. An attempt was also made to validate thefour quality dimensions and the sum of the fourdimensions by investigating predictors of qualityscores. Study design, intervention and theinteraction of study design and intervention wereentered into the regression model. For totalquality scores, both cohort and case–controlstudies had significantly poorer quality than RCTs.There was no independent effect of interventionand no interaction of intervention and studydesign. Case–control studies scored more highlythan cohort studies.For the 71 tools developed by review authors, nodetails of the tool development were provided andnone justified the choice of quality items. Twentyonereviews examined inter-rater reliability andfound similar levels of agreement to thosediscussed previously. In addition, two furtherreviews 93,129 addressed the validity of their tools.45© Queen’s Printer and Controller of HMSO 2003. All rights reserved.