Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27results of both observational and randomisedstudies. It seems highly likely that these decisionsmay relate to the similarity of the results of studiesof different designs.Did the RCTs and non-randomised studiesrecruit similar participants, use similarinterventions and measure similar outcomes?Discrepancies between the results of observationaland randomised studies may be confounded bydifferences in the selection and evaluation ofpatient groups, in the formulation and delivery oftreatments, in the use of placebos and othercomparative treatments and in the methods usedto maintain follow-up and record outcomes. Formany interventions there may also be temporalconfounding of study types, non-randomisedstudies typically being performed prior to theRCTs. Such meta-confounding will make it difficultto attribute a systematic difference directly to theuse or non-use of a random allocation mechanism.Six of the eight reviews noted this problem andincorporated features in their evaluations toreduce the potential for metaconfounding.25–28,32,35 Two reviews made morestringent efforts to assess comparability than theothers. 25,26 Britton and colleagues restricted theselection of studies to be similar in terms ofintervention, setting, control therapy and outcomemeasure. MacLehose and colleagues assessed eachcomparison for the possibility of metaconfoundingand found that the most susceptible(i.e. those with differences in eligibility criteria andtime periods and no adjustment for severity ofdisease, comorbidity and other prognostic factors)had, on average, the largest discrepancies.Were the RCTs and non-randomised studiesshown to use similar study methodology in allrespects other than the allocation mechanism?Meta-confounding could also occur throughdifferences in other aspects of study design,beyond the use of randomisation. For example,the results of RCTs are known to vary according tothe quality of randomisation, especially theconcealment of allocation at recruitment. 23However, none of the reviews restricted theinclusion of RCTs to those with adequateconcealment or on any other methodologicalbasis. Only one review 26 assessed the comparabilityof randomised and non-randomised studies onany aspect of study quality (blinding).Discrepancies and similarities between studydesigns could be partly explained by differences inother unevaluated aspects of the methodologicalquality of the RCTs.Similarly, there will be differences in themethodological rigour of the non-randomisedstudies. Importantly, the possible biases of nonrandomisedstudies vary with study design. Onlyone review 27 restricted non-randomised studies tobe of a single design (historically controlledstudies). In all the others, RCTs were comparedwith non-randomised studies of a mixture ofdesigns.Were sensible, objective criteria used todetermine differences or equivalence of studyfindings?The manner in which results were judged to be‘equivalent’ or ‘discrepant’ varied widely betweenthe reviews and influenced the conclusions thatwere drawn. For example, Concato andcolleagues 33 deemed that the randomised andnon-randomised studies of mammographicscreening (comparison 34 in Table 3) hadremarkably similar results, whereas Ioannidis andcolleagues 34 classified them as discrepant. Only intwo reviews was the judgement made aggregatedacross the comparisons. 27,35 In all the otherreviews each individual topic was classified aseither equivalent or discrepant. Many of thecomparisons made at the level of a clinical topicwere based on very few data, for example, in theIoannidis review 34 on average for eachintervention five RCTs were compared with fournon-randomised studies. Hence the absence of astatistically significant difference cannot beinterpreted as evidence of ‘equivalency’ andclinically significant differences in treatmenteffects cannot be excluded. Conversely, thepresence of a statistically significant differencedoes not indicate that a clinically importantdifference does exist. Four reviews 26,28,34,35 moreusefully concentrated on describing the magnitudeof the differences, all four noting substantialdifferences occurring in some, but not all,comparisons. The Concato review 33 subjectivelyclassified all comparisons as being ‘remarkablysimilar’.The comparisons made in two reviews 33,34 of therelative variability of randomised and nonrandomisedresults can be considered flawedowing to the criteria used to compare variation.Concato and colleagues considered the range ofthe point estimates from observational studies andRCTs. 33 This comparison was confounded by thedifferent sample sizes used in observational andrandomised studies, and the number of studiesconsidered. On average, the RCTs in the Concatoreview were 25% smaller than the observationalstudies, hence greater variability in their results is19© Queen’s Printer and Controller of HMSO 2003. All rights reserved.