Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27explanation of responses to aid completion. Theconcept behind the tool seems a good one, but itis difficult to complete in isolation from the rest ofthe data extraction form. Despite the instructionsfor completion, some of the items may be toogeneric and difficult to interpret. For example,allocation of participants to groups is consideredin the ‘interpretation of results’ section under theitem ‘considering the study design, wereappropriate methods for controlling confoundingvariables and limiting potential biases used’. Thisitem aimed to cover randomisation, restriction,matching, etc. The complexity of the tool meantthat it took up to 30 minutes to complete. It maybe that the ease of use would increase withpractice. The Zaza tool may be suitable for use insystematic reviews but does require a goodunderstanding of validity issues.DiscussionA total of 213 potential quality assessment toolswere identified for inclusion in this review. Overallthe tools were poorly developed, with almost noattention to standard principles of scaledevelopment. 57 Almost without exception, theitems included in the tools were based on so-called‘standard criteria’ gleaned from methodologicalliterature, clinical trial or epidemiology textbooksor a review of other quality assessment tools. Mosttools did not provide a means of assessing theinternal validity of non-randomised studies, andseveral were aimed specifically at only randomisedtrials. Only 60 (30%) included items related to fiveout of six internal validity domains. Of these, 14were sufficiently comprehensive in their contentcoverage to be considered in detail. In order to beselected, these tools had to include at least threeof four pre-specified core items:●●●●how allocation occurredany attempt to balance groups by designidentification of prognostic factorscase-mix adjustment.These were selected as core items because for nonrandomisedstudies it is important to know, first,how study participants got into the interventiongroups, that is, was it by clinician or patientpreference, or was it according to spatial ortemporal factors and, second, what specific factorsinfluenced the selection of patients into eachgroup. Given that study investigators rarely reportthe latter information, the identification ofprognostic factors that influence outcome (asopposed to allocation) was included as a core item.However, covering these items does not necessarilymake the tools useful. For example, asking for adescription of the method of allocation 109,111 doesnot force a judgement about whether that methodwas appropriate or unlikely to introduce bias.Other than their content coverage, the top 14tools did not stand out from the remaining toolsin terms of their development, which was oftenvaguely reported, or the investigation of validityor reliability.A relatively informal assessment of the usefulnessof the top 14 tools identified six that werepotentially useful for systematicreviews. 65,66,85,86,109,111 The main advantage ofthese tools over the remaining eight was thephrasing of the items. On the whole, these forcethe reviewer to be systematic in their studyassessments and attempt to ensure that qualityjudgements are made in the most objectivemanner possible. Four of the tools attempted tocover most study designs using the samequestions, which could be the most usefulapproach for a systematic review whichincorporates several different study designs. Thisapproach did appear reasonably successful,although it may often be more appropriate tothink of quality issues on top of a study design‘hierarchy’. For example, there seems to be littlepoint in being overly concerned with blindinglevels when comparing an RCT with a before-andafterstudy. Furthermore, in many cases a fullassessment of study quality may require additionalcontext-specific questions to cover aspects ofexternal validity that would not be included in ageneric quality assessment tool, for example, itemsrelating to the quality of delivery of anintervention or quality of outcome measurements.Many of the tools, including some in the top14, 85,111 contained several items unrelated tomethodological quality. Although it is important todistinguish the quality of a study from the qualityof its report, one of the identified problems withscales for assessing RCTs is the inclusion of itemsrelating to issues such as reporting quality. 59Those tools which followed the lines of Cooper’s‘mixed-criteria’ approach, 45 requiring objectivefacts regarding a study’s design followed by aquality judgement (e.g. Thomas 65 ), may prove tobe the most useful for systematic reviews. Suchtools make as explicit as possible the factsregarding a study’s methods that should underliea reviewer’s judgement. Some tools were foundthat ignore judgements entirely and others41© Queen’s Printer and Controller of HMSO 2003. All rights reserved.