Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27MethodologyEvaluating non-randomisedintervention studiesJJ DeeksJ DinnesR D’AmicoAJ SowdenC SakarovitchF SongM PetticrewDG AltmanHealth Technology AssessmentNHS R&D HTA ProgrammeHTA

HTAHow to obtain copies of this and other HTA Programme reports.An electronic version of this publication, in Adobe Acrobat format, is available for downloading free ofcharge for personal use from the HTA website (http://www.hta.ac.uk). A fully searchable CD-ROM isalso available (see below).Printed copies of HTA monographs cost £20 each (post and packing free in the UK) to both public andprivate sector purchasers from our Despatch Agents.Non-UK purchasers will have to pay a small fee for post and packing. For European countries the cost is£2 per monograph and for the rest of the world £3 per monograph.You can order HTA monographs from our Despatch Agents:– fax (with credit card or official purchase order)– post (with credit card or official purchase order or cheque)– phone during office hours (credit card only).Additionally the HTA website allows you either to pay securely by credit card or to print out yourorder and then post or fax it.Contact details are as follows:HTA DespatchEmail: orders@hta.ac.ukc/o Direct Mail Works Ltd Tel: 02392 492 0004 Oakwood Business Centre Fax: 02392 478 555Downley, HAVANT PO9 2NP, UK Fax from outside the UK: +44 2392 478 555NHS libraries can subscribe free of charge. Public libraries can subscribe at a very reduced cost of£100 for each volume (normally comprising 30–40 titles). The commercial subscription rate is £300per volume. Please see our website for details. Subscriptions can only be purchased for the current orforthcoming volume.Payment methodsPaying by chequeIf you pay by cheque, the cheque must be in pounds sterling, made payable to Direct Mail Works Ltdand drawn on a bank with a UK address.Paying by credit cardThe following cards are accepted by phone, fax, post or via the website ordering pages: Delta, Eurocard,Mastercard, Solo, Switch and Visa. We advise against sending credit card details in a plain email.Paying by official purchase orderYou can post or fax these, but they must be from public bodies (i.e. NHS or universities) within the UK.We cannot at present accept purchase orders from commercial companies or from outside the UK.How do I get a copy of HTA on CD?Please use the form on the HTA website (www.hta.ac.uk/htacd.htm). Or contact Direct Mail Works (seecontact details above) by email, post, fax or phone. HTA on CD is currently free of charge worldwide.The website also provides information about the HTA Programme and lists the membership of the variouscommittees.

Evaluating non-randomisedintervention studiesJJ Deeks 1*J Dinnes 2R D’Amico 1AJ Sowden 3C Sakarovitch 1F Song 4M Petticrew 5DG Altman 1In collaboration with the International Stroke Trial and the EuropeanCarotid Surgery Trial Collaborative Groups1 Centre for Statistics in Medicine, Institute of Health Sciences, Oxford, UK2 Southampton Health Technology Assessments Centre, University ofSouthampton, UK3 NHS Centre for Reviews and Dissemination, University of York, UK4 Department of Public Health and Epidemiology, University ofBirmingham, UK5 MRC Social and Public Health Sciences Unit, University of Glasgow, UK* Corresponding authorDeclared competing interests of authors: nonePublished September 2003This report should be referenced as follows:Deeks JJ, Dinnes J, D’Amico R, Sowden AJ, Sakarovitch C, Song F, et al. Evaluatingnon-randomised intervention studies. Health Technol Assess 2003;7(27).Health Technology Assessment is indexed in Index Medicus/MEDLINE and Excerpta Medica/EMBASE.

NHS R&D HTA ProgrammeThe NHS R&D Health Technology Assessment (HTA) Programme was set up in 1993 to ensurethat high-quality research information on the costs, effectiveness and broader impact of healthtechnologies is produced in the most efficient way for those who use, manage and provide carein the NHS.Initially, six HTA panels (pharmaceuticals, acute sector, primary and community care, diagnosticsand imaging, population screening, methodology) helped to set the research priorities for the HTAProgramme. However, during the past few years there have been a number of changes in and aroundNHS R&D, such as the establishment of the National Institute for Clinical Excellence (NICE) andthe creation of three new research programmes: Service Delivery and Organisation (SDO); New andEmerging Applications of Technology (NEAT); and the Methodology Programme.The research reported in this monograph was identified as a priority by the HTA Programme’sMethodology Panel and was funded as project number 96/26/99.The views expressed in this publication are those of the authors and not necessarily those of theMethodology Programme, HTA Programme or the Department of Health. The editors wish toemphasise that funding and publication of this research by the NHS should not be taken as implicitsupport for any recommendations made by the authors.Criteria for inclusion in the HTA monograph seriesReports are published in the HTA monograph series if (1) they have resulted from workcommissioned for the HTA Programme, and (2) they are of a sufficiently high scientific qualityas assessed by the referees and editors.Reviews in Health Technology Assessment are termed ‘systematic’ when the account of the search,appraisal and synthesis methods (to minimise biases and random errors) would, in theory, permitthe replication of the review by others.Methodology Programme Director: Professor Richard LilfordHTA Programme Director: Professor Kent WoodsSeries Editors:Professor Andrew Stevens, Dr Ken Stein, Professor John Gabbay,Dr Ruairidh Milne and Dr Rob RiemsmaManaging Editors:Sally Bailey and Sarah Llewellyn LloydThe editors and publisher have tried to ensure the accuracy of this report but do not accept liabilityfor damages or losses arising from material published in this report. They would like to thank thereferees for their constructive comments on the draft document.ISSN 1366-5278© Queen’s Printer and Controller of HMSO 2003This monograph may be freely reproduced for the purposes of private research and study and may be included in professional journalsprovided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising.Applications for commercial reproduction should be addressed to HMSO,The Copyright Unit, St Clements House,2–16 Colegate, Norwich, NR3 1BQ.Published by Gray Publishing, Tunbridge Wells, Kent, on behalf of NCCHTA.Printed on acid-free paper in the UK by St Edmundsbury Press Ltd, Bury St Edmunds, Suffolk.M

Health Technology Assessment 2003; Vol. 7: No. 27AbstractEvaluating non-randomised intervention studiesJJ Deeks, 1* J Dinnes, 2 R D’Amico, 1 AJ Sowden, 3 C Sakarovitch, 1 F Song, 4M Petticrew 5 and DG Altman 1In collaboration with the International Stroke Trial and the European Carotid Surgery Trial Collaborative Groups1 Centre for Statistics in Medicine, Institute of Health Sciences, Oxford, UK2 Southampton Health Technology Assessments Centre, University of Southampton, UK3 NHS Centre for Reviews and Dissemination, University of York, UK4 Department of Public Health and Epidemiology, University of Birmingham, UK5 MRC Social and Public Health Sciences Unit, University of Glasgow, UK* Corresponding authorObjectives: To consider methods and related evidencefor evaluating bias in non-randomised interventionstudies.Data sources: Systematic reviews and methodologicalpapers were identified from a search of electronicdatabases; handsearches of key medical journals andcontact with experts working in the field. Newempirical studies were conducted using data from twolarge randomised clinical trials.Methods: Three systematic reviews and new empiricalinvestigations were conducted. The reviews considered,in regard to non-randomised studies, (1) the existingevidence of bias, (2) the content of quality assessmenttools, (3) the ways that study quality has been assessedand addressed. (4) The empirical investigations wereconducted generating non-randomised studies fromtwo large, multicentre randomised controlled trials(RCTs) and selectively resampling trial participantsaccording to allocated treatment, centre andperiod.Results: In the systematic reviews, eight studiescompared results of randomised and non-randomisedstudies across multiple interventions using metaepidemiologicaltechniques. A total of 194 tools wereidentified that could be or had been used to assessnon-randomised studies. Sixty tools covered at leastfive of six pre-specified internal validity domains.Fourteen tools covered three of four core items ofparticular importance for non-randomised studies. Sixtools were thought suitable for use in systematicreviews. Of 511 systematic reviews that included nonrandomisedstudies, only 169 (33%) assessed studyquality. Sixty-nine reviews investigated the impact ofquality on study results in a quantitative manner. Thenew empirical studies estimated the bias associatedwith non-random allocation and found that the biascould lead to consistent over- or underestimations oftreatment effects, also the bias increased variation inresults for both historical and concurrent controls,owing to haphazard differences in case-mix betweengroups. The biases were large enough to lead studiesfalsely to conclude significant findings of benefit orharm. Four strategies for case-mix adjustment wereevaluated: none adequately adjusted for bias inhistorically and concurrently controlled studies. Logisticregression on average increased bias. Propensity scoremethods performed better, but were not satisfactory inmost situations. Detailed investigation revealed thatadequate adjustment can only be achieved in theunrealistic situation when selection depends on a singlefactor.Conclusions: Results of non-randomised studiessometimes, but not always, differ from results ofrandomised studies of the same intervention. Nonrandomisedstudies may still give seriously misleadingresults when treated and control groups appear similarin key prognostic factors. Standard methods of case-mixadjustment do not guarantee removal of bias. Residualconfounding may be high even when good prognosticdata are available, and in some situations adjustedresults may appear more biased than unadjusted results.Although many quality assessment tools exist and havebeen used for appraising non-randomised studies, mostomit key quality domains. Healthcare policies basedupon non-randomised studies or systematic reviews ofnon-randomised studies may need re-evaluation if theuncertainty in the true evidence base was not fullyappreciated when policies were made. The inability ofiii© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Abstractcase-mix adjustment methods to compensate forselection bias and our inability to identify nonrandomisedstudies that are free of selection biasindicate that non-randomised studies should only beundertaken when RCTs are infeasible or unethical.Recommendations for further research include: applyingthe resampling methodology in other clinical areas toascertain whether the biases described are typical;developing or refining existing quality assessment toolsfor non-randomised studies; investigating how qualityassessments of non-randomised studies can beincorporated into reviews and the implications ofindividual quality features for interpretation of areview’s results; examination of the reasons for theapparent failure of case-mix adjustment methods; andfurther evaluation of the role of the propensity score.iv

Health Technology Assessment 2003; Vol. 7: No. 27ContentsList of abbreviations ..................................Executive summary ....................................1 Introduction ............................................... 1Types of non-randomised studies .............. 1Sources of bias in non-randomisedstudies ......................................................... 3Case-mix adjustment methods ................... 5Scope of this project ................................... 52 Aims and objectives ................................... 73 Review of empirical comparisons ofthe results of randomised andnon-randomised studies ............................ 9Introduction ............................................... 9Methods ...................................................... 9Results ........................................................ 10Discussion ................................................... 204 Evaluation of checklists and scales forassessing quality of non-randomisedstudies ........................................................ 23Introduction ............................................... 23Methods ...................................................... 24Results ........................................................ 29Discussion ................................................... 415 Use of quality assessment in systematicreviews of non-randomised studies ........... 43Introduction ............................................... 43Methods ...................................................... 43Results ........................................................ 44Discussion ................................................... 476 Empirical estimates of bias associated withnon-random allocation ............................... 49Introduction ............................................... 49Methods ...................................................... 49Results ........................................................ 52Discussion ................................................... 57Conclusions ................................................ 62viiix7 Empirical evaluation of the ability ofcase-mix adjustment methodologies tocontrol for selection bias ........................... 63Introduction ............................................... 63Methods ...................................................... 65Results ........................................................ 71Discussion ................................................... 78Conclusions ................................................ 858 Discussion and conclusions ........................ 87Summary of key findings ........................... 87Discussion ................................................... 88Conclusions ................................................ 91Acknowledgements .................................... 93References .................................................. 95Appendix 1 Search strategy forChapters 3–5 .............................................. 111Appendix 2 Data extraction forms ............ 113Appendix 3 Details of identified qualityassessment tools ......................................... 121Appendix 4 Detailed description of ‘unselected’top 14 quality assessment tools .................... 135Appendix 5 Coverage of core items byselected top tools ........................................ 139Appendix 6 Details of review methods ..... 141Appendix 7 Results of identified systematicreviews ........................................................ 151Appendix 8 Descriptions of the IST andECST .......................................................... 167Health Technology and Assessment reportspublished to date ....................................... 175Health Technology and AssessmentProgramme ................................................ 183v

Health Technology Assessment 2003; Vol. 7: No. 27List of abbreviationsBCGbacille Calmette–Guérin vaccineMAmeta-analysisCASPCHDCritical Appraisal SkillsProgrammecoronary heart diseaseMIMSK CRGmyocardial infarctionCochrane MusculoskeletalCollaborative Review GroupCINAHLCumulative Index to Nursing andAllied Health LiteratureNASCETNorth American SymptomaticCarotid Endarterectomy TrialCTCVDAREcomputed tomographycovariateDatabase of Abstracts of Reviewsof EffectivenessNRSORPSQAnon-randomised studyodds ratiopropensity scorequality assessmentECSTEuropean Carotid Surgery TrialRCTrandomised controlled trialERICEducational ResourcesInformation Centre databaseRRSDrisk ratiostandard deviationHCThistorically controlled trialSEstandard errorISTInternational Stroke TrialTBtuberculosislog ORlogarithm of the oddsratioTENStranscutaneous electrical nervestimulationLRlogistic regressionTIAtransient ischaemic attackAll abbreviations that have been used in this report are listed here unless the abbreviation is well known (e.g. NHS), orit has been used only once, or it is a non-standard abbreviation used only in figures/tables/appendices in which casethe abbreviation is defined in the figure legend or at the end of the table.vii© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Health Technology Assessment 2003; Vol. 7: No. 27Executive summaryBackgroundIn the absence of randomised controlled trials(RCTs), healthcare practitioners and policy-makersrely on non-randomised studies to provideevidence of the effectiveness of healthcareinterventions. However, there is controversyover the validity of non-randomised evidence,related to the existence and magnitude ofselection bias.ObjectivesTo consider methods and related evidence forevaluating bias in non-randomised interventionstudies.Methods1. Three reviews were conducted to consider:● empirical evidence of bias associated withnon-randomised studies● the content of quality assessment tools fornon-randomised studies● the use of quality assessment in systematicreviews of non-randomised studies.These reviews were conducted systematically,identifying relevant literature throughcomprehensive searches across electronicdatabases, handsearches and contact withexperts.2. New empirical investigations were conductedgenerating non-randomised studies from twolarge, multicentre RCTs by selectivelyresampling trial participants according toallocated treatment, centre and period. Thesewere used to examine:● systematic bias introduced by the use ofhistorical and non-randomised concurrentcontrols● whether results of non-randomised studiesare more variable than results of RCTs● the ability of case-mix adjustment methods tocorrect for selection bias introduced by nonrandomallocation.The resampling design overcame particularproblems of meta-confounding and variability ofdirection and magnitude of bias that hinder theinterpretation of previous reviews.ResultsEmpirical comparisons of randomisedand non-randomised evidenceEight studies compared results of randomised andnon-randomised studies across multipleinterventions using meta-epidemiologicaltechniques. The studies reached conflictingconclusions, explicable by differences in:●●●●whether data were sourced from primary studiesor systematic reviewsconsideration of meta-confoundinginclusion of studies of varying qualitycriterion for classifying discrepancies inresults.The only deducible conclusions were(a) results of randomised and non-randomisedstudies sometimes, but not always, differ and(b) both similarities and differences mayoften be explicable by other confoundingfactors.Quality assessment tools for evaluatingnon-randomised studiesWe identified 194 tools that could be or had beenused to assess non-randomised studies. Aroundhalf were scales and half checklists, most werepublished within systematic reviews and most werepoorly developed with scant attention paid toprinciples of scale development.Sixty tools covered at least five of six pre-specifiedinternal validity domains (creation of groups,blinding, soundness of information, follow-up,analysis of comparability, analysis of outcome),although the degree of coverage varied.Fourteen tools covered three of four core itemsof particular importance for non-randomisedstudies (How allocation occurred? Was thestudy designed to generate comparablegroups? Were prognostic factors identified?Was case-mix adjustment used?). Six toolswere thought suitable for use in systematicreviews.ix© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Executive summaryUse of quality assessment in systematicreviews of non-randomisedstudiesOf 511 systematic reviews that included nonrandomisedstudies, only 169 (33%) assessed studyquality. Many used quality assessment toolsdesigned for RCTs or developed by the authorsthemselves, and did not include key qualitycriteria relevant to non-randomised studies. Sixtyninereviews investigated the impact of quality onstudy results in a quantitative manner.Empirical estimates of bias associatedwith non-random allocationThe bias introduced by non-random allocation wasnoted to have two components. First, the biascould lead to consistent over- or underestimationsof treatment effects. This occurred for historicalcontrols, the direction of bias depending on timetrends in the case-mix of participants recruited tothe study. Second, the bias increased variation inresults for both historical and concurrent controls,owing to haphazard differences in case-mixbetween groups. The biases were large enough tolead studies falsely to conclude significant findingsof benefit or harm.Empirical evaluation of case-mixadjustment methodsFour strategies for case-mix adjustment wereevaluated: none adequately adjusted for bias inhistorically and concurrently controlled studies.Logistic regression on average increased bias.Propensity score methods performed better, butwere not satisfactory in most situations. Detailedinvestigation revealed that adequate adjustmentcan only be achieved in the unrealistic situationwhen selection depends on a single factor.Omission of important confounding factors canexplain underadjustment. Correlatedmisclassifications and measurement error inconfounding variables may explain the observedincrease in bias with logistic regression, as maydifferences between conditional and unconditionalodds ratio estimates of treatment effects.ConclusionsResults of non-randomised studies sometimes, butnot always, differ from results of randomisedstudies of the same intervention. Non-randomisedstudies may still give seriously misleading resultswhen treated and control groups appear similar inkey prognostic factors. Standard methods of casemixadjustment do not guarantee removal of bias.Residual confounding may be high even whengood prognostic data are available, and in somesituations adjusted results may appear more biasedthan unadjusted results.Although many quality assessment tools exist andhave been used for appraising non-randomisedstudies, most omit key quality domains. Six toolswere considered potentially suitable for use insystematic reviews, but each requires revision tocover all relevant quality domains.Healthcare policies based upon non-randomisedstudies or systematic reviews of non-randomisedstudies may need re-evaluation if the uncertaintyin the true evidence base was not fully appreciatedwhen policies were made.The inability of case-mix adjustment methods tocompensate for selection bias and our inability toidentify non-randomised studies which are free ofselection bias indicate that non-randomisedstudies should only be undertaken when RCTs areinfeasible or unethical.Recommendations for furtherresearch●●●●●The resampling methodology utilised hereshould be applied in other clinical areas toascertain whether the biases we describe aretypical.Efforts should be focused on developing qualityassessment tools for non-randomised studies,possibly by refining existing tools.Research should consider how qualityassessment of non-randomised studies can beincorporated into reviews, and the implicationsof individual quality features for interpretationof a review’s results.Reasons for the apparent failure of case-mixadjustment methods should be furtherinvestigated, including assessments of thegeneralisability of our results to risk assessmentsand epidemiological studies, and differencesbetween conditional and unconditionalestimates of treatment effects.The role of the propensity score should befurther evaluated, and computer macros madeavailable for its application.x

Health Technology Assessment 2003; Vol. 7: No. 27Chapter 1IntroductionThe ultimate goal of the evaluation ofhealthcare interventions is to produce a validestimate of effectiveness, in terms of both internaland external validity. Internal validity concerns theextent to which the results of a study can bereliably attributed to the intervention underevaluation, whereas external validity concerns theextent to which a study’s results can be generalisedbeyond the given study context.The randomised controlled trial (RCT) is widelyregarded as the design of choice for theassessment of the effectiveness of healthcareinterventions. The main benefit of the RCT is theuse of a randomisation procedure that, whenproperly implemented, ensures that the allocationof any participant to one treatment or anothercannot be predicted. The randomisation processmakes the comparison groups equal with respectto both known and unknown prognostic factors atbaseline, apart from chance bias. 1 RCTs also tendto benefit from so-called ‘inherited properties’,which generally mark them out as higher qualitystudies. These properties include the fact that theyare prospective studies, with written protocolsspecifying, and thus standardising, importantaspects of patient enrolment, treatment,observation and analysis. 2 RCTs are also morelikely to employ specific measures to reduce orremove bias, such as blinded outcome assessment.There are instances where non-randomised studieshave either been sufficient to demonstrateeffectiveness or they appear to have arrived atresults similar to those of RCTs. However, whererandomisation is possible, most agree that theRCT should be the preferred method ofevaluating effectiveness. 2–4 The risks of relyingsolely on non-randomised evidence include failingto convince some people of the validity of theresult, or successfully convincing others of anincorrect result. 3Nevertheless, several scenarios remain underwhich an RCT may be unnecessary, inappropriate,impossible or inadequate. 5 Examples include theassessment of rare side-effects of treatments, somepreventive interventions and policy changes.Furthermore, there must be hundreds of examplesof interventions for which RCTs would be possiblebut have not yet been carried out, leaving themedical and policy community to rely on nonrandomisedevidence. It is therefore essential tohave an understanding of the biases that mayinfluence non-randomised studies.Types of non-randomised studiesA taxonomy of study designs that may be used toassess the effectiveness of an intervention isprovided in Box 1. However, there is inconsistentuse of nomenclature when describing nonrandomisedstudies, and other taxonomies mayapply different definitions to the same studydesigns. To attempt to avoid the problems ofinconsistent terminology, six features can beidentified that differentiate between these studies.First, some studies make comparisons betweengroups, whilst some simply describe outcomes in asingle group (e.g. case series). Second, thecomparative designs differ in the way thatparticipants are allocated to groups, varying fromthe use of randomisation (RCTs), quasirandomisation,geographical or temporal factors(cohort studies), the decisions of healthcareprofessionals (clinical database cohorts), to theidentification of groups with specific outcomes(case–control studies). Third, studies differ in thedegree to which they are prospective (andtherefore planned) or retrospective, for matterssuch as the recruitment of participants, collectionof baseline data, collection of outcome data andgeneration of hypotheses. Fourth, the methodused to investigate comparability of the groupsvaries: in RCTs no investigation is necessary(although it is often carried out), in controlledbefore-and-after designs baseline outcomemeasurements are used, and in cohort andcase–control studies investigation of confoundersis required. Fifth, studies differ in the level atwhich the intervention is applied: sometimes it isallocated to individuals, other times to groups orclusters.Finally, some studies are classified as experimentalwhereas others are observational. In experimentalstudies the study investigator has some degree ofcontrol over the allocation of interventions. Mostimportantly, he/she has control over the allocation1© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

IntroductionBOX 1 Taxonomy of study designs to assess the effectiveness of an interventionExperimental designsA study in which the investigator has control over at least some study conditions, particularly decisions concerning theallocation of participants to different intervention groups.1. Randomised controlled trialParticipants are randomly allocated to intervention or control groups and followed up over time to assess any differencesin outcome rates. Randomisation with allocation concealment ensures that on average known and unknowndeterminants of outcome are evenly distributed between groups.2. Quasi-randomised trialParticipants are allocated to intervention or control groups by the investigator, but the method of allocation falls short ofgenuine randomisation and allocation concealment (e.g. allocated by date of birth, hospital record number, etc.)3. Non-randomised trial/quasi-experimental studyThe investigator has control over the allocation of participants to groups, but does not attempt randomisation (e.g.patient or physician preference). Differs from a ‘cohort study’ in that the intention is experimental rather thanobservational.Observational designsA study in which natural variation in interventions (or exposure) among study participants is investigated to explore the effectof the interventions (or exposure) on health outcomes.4. Controlled before-and-after studyA follow-up study of participants who have received an intervention and those who have not, measuring the outcomevariable both at baseline and after the intervention period, comparing either final values if the groups are comparable atbaseline, or change scores. It can also be considered an experimental design if the investigator has control over, or candeliberately manipulate, the introduction of the intervention.5. Concurrent cohort studyA follow-up study that compares outcomes between participants who have received an intervention and those who havenot. Participants are studied during the same (concurrent) period either prospectively or, more commonly,retrospectively.6. Historical cohort studyA variation on the traditional cohort study where the outcome from a new intervention is established for participantsstudied in one period and compared with those who did not receive the intervention in a previous period, i.e.participants are not studied concurrently.7. Case–control studyParticipants with and without a given outcome are identified (cases and controls respectively) and exposure to a givenintervention(s) between the two groups compared.8. Before-and-after studyComparison of outcomes from study participants before and after an intervention is introduced. The before and aftermeasurements may be made in the same participants, or in different samples. It can also be considered an experimentaldesign if the investigator has control over, or can deliberately manipulate, the introduction of the intervention.9. Cross-sectional studyExamination of the relationship between disease and other variables of interest as they exist in a defined population atone particular time point.10. Case seriesDescription of a number of cases of an intervention and outcome (no comparison with a control group).Adapted from CRD Report 4. 1752of participants to intervention groups either usingrandomisation of participants, or haphazardallocation by alternation, dates of birth, day of theweek or case record numbers. The key to a wellcontrolledexperimental study is the concealmentof the allocation schedule from the studyinvestigators, such that the allocation of any givenparticipant cannot be predicted. 6 When this is notensured, major bias may be introduced. Inobservational studies, on the other hand, thegroups that are compared are generated accordingto variation in the use of interventions that occursregardless of the study. When allocation isdetermined largely by health professionals, thetreatment decision is based not only on ‘hard’ datasuch as age, sex and diagnostic test results, butalso on ‘soft’ data, including type and severity ofsymptoms, rate of development of the illness, andseverity of any co-morbid conditions, which arerarely made explicit. 7 Allocation in observational

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 1 Sources of biasSource of bias RCTs Cohort studiesSelection bias Randomisation Control for confoundersPerformance bias Blinding (of participants and/or investigators) Measurement of exposureAttrition bias Completeness of follow-up Completeness of follow-upDetection bias Blinded outcome assessment Blinded outcome assessmentstudies may also be based on factors such asavailability of care or geographical location. Inobservational studies, therefore, there are likely tobe systematic differences in the case-mix ofpatients allocated to the intervention andcomparison groups.Allocation to groups can also be based on patientchoice, as in patient preference trials. 8 Individualpreferences for one treatment above another maywell imply differences in other consistent ways(potentially relating to prognosis), from those whodo not hold such a preference. 9 In addition,preference for a particular treatment may enhanceits therapeutic effect.Sources of bias in nonrandomisedstudiesThe Cochrane Collaboration handbook has laidout the four main sources of systematic bias intrials of the effects of healthcare as being selectionbias, performance bias, attrition bias and detectionbias (Table 1). All of these biases can affect nonrandomisedstudies and are discussed in thisreport. However, it is selection bias that we discussin the greatest detail, and evaluate in newempirical studies, as it is the potential for selectionbias that most clearly differentiates randomisedfrom non-randomised studies.Selection biasThe greatest distinction between the results ofrandomised and non-randomised studies is, asdescribed above, the risk of selection bias, wheresystematic differences in comparison groups ariseat baseline. The term selection bias can bemisleading as it is used to describe both biasedselection of participants for inclusion in a study(which applies to both experimental andobservational studies) and biased allocation ofpatients to a given intervention (which occurswhere randomisation is not used). The first type ofselection bias is usually classified as an issue ofexternal validity and is not discussed further inthis report. Rather, we consider the second type,which is an issue of internal validity. It is sometimesreferred to as case-mix bias, or confounding.In non-randomised studies, selection bias will beintroduced when participants chosen for oneintervention have different characteristics fromthose allocated to the alternative intervention (ornot treated). For observational studies the choiceof a given intervention is largely at the discretionof the treating clinician (as would occur in normalclinical practice). The choice of an interventionunder these circumstances will be influenced notonly by a clinician’s own personal preference forone intervention over another but also by patientpreference, patient characteristics and clinicalhistory. Sometimes the reasons for the choice of atreatment will be obvious, but at other times aclinician’s treatment decision will be influenced bysubtle clues that are not easily identifiable. 3 Thismay result in treatment groups that areincomparable, often with one intervention group‘heavily weighted by the more severely ill’. 10According to Miettinen, 11 in clinical practice (andtherefore in observational studies):“Interventions are commonly prompted by anindication, a state or event that signifies the prospectof an untoward outcome. Thus, by the very rationalityof decisions to intervene, the treated tend to differfrom the untreated with respect to their outlooks forthe outcome criterion in efficacy assessment; theretends to be confounding by the indication – usually suchthat the treated tend to have less favourable outcomethan the untreated.”In other words, when faced with a patient whomay be eligible to receive a given intervention,the decision to treat will be influenced by somefactor that in turn is related to the treatmentoutcome. This introduces systematic bias leadingto either over- or underestimates of treatmenteffects, depending on the treatment decisionmechanism.Confounding by indication can take several guisesand the term has been used to describe a numberof situations. The original definition refers to thesituation where an extraneous condition is both a3© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Introduction4clinical indication for the application of theintervention under study and an indication for theoutcome under study. For example, arrestedlabour is a factor that may directly contribute tothe decision to introduce fetal monitoring, that is,arrested labour is an indication for the monitoringprocedure. Any observational study would beexpected to show a higher proportion of arrestedlabours among the monitored participants. Sincearrested labour is also an indication for Caesareansection, we would expect a higher Caesareanfrequency among monitored labours, even ifmonitoring had no effect. 12 Where the indication(arrested labour) influences the use of theintervention (fetal monitoring) and is a risk factorfor the outcome (Caesarean section), there will bean imbalance in prognosis between the treatmentgroups being compared.Confounding by severity (or by prognosis) may beconsidered as a special form of confounding byindication. It occurs where the severity of theextraneous condition influences both theintroduction of the intervention and the outcomeunder study, 13 that is, any treatment reserved forthe most ill will be associated with the worstoutcomes (even where the treatment is effective).The underlying deterioration of the treateddisease predicts the use of a given treatment,which in turn becomes associated with theprogression of the disease. 14 For example, thefrequent use of β-agonists predicts death fromasthma because those with most severe asthma aremore likely to be prescribed β-agonists.Protopathic bias is a term coined by Horwitzand Feinstein 15 to describe situations where thefirst symptoms of a given outcome are the reasonfor treatment initiation: “Protopathic bias”occurs “when a pharmaceutical or othertherapeutic agent is inadvertently prescribed foran early manifestation of a disease that has not yetbeen diagnostically detected” (our emphasis). Forexample, a drug given for abdominal pain may bewrongly associated with hepatic injury, asabdominal pain may be one of the prodromalsymptoms. 13Unfortunately, in practice, the ‘subtle clues’prompting the selection of a particularintervention for a given patient are oftenunrecognised and unrecorded. When discussingthe use of observational databases, Byar 16 pointedout that “I have yet to see an analysis presentedtoday (or in my whole life for that matter) thathad really good information on why some patientsgot one treatment and others got another”.One approach to improving the design of nonrandomisedstudies is the ‘restricted cohort’ designfirst developed by Horwitz and Feinstein 15 and laterrefined by the same group. 17 This approach involvesrestricting the eligibility criteria of cohort studies tothose used in clinical trials, defining a ‘zero-time’point from which patients are followed up, andusing an approximation of intention-to-treatanalysis. They have demonstrated, for exampleusing a study of β-blocker therapy after acutemyocardial infarction, 17 that similar results could beobtained to those reported by RCTs. This approachdoes not appear to have been widely taken up, andthe original proposal has received some criticism. 3The degree to which non-random allocationmethods are susceptible to selection bias andtherefore may produce biased estimates of theeffect of treatment is not clearly understood,although it seems likely that the potential for biaswill vary between clinical areas. It is reasonable toexpect that the comparability of the groups interms of prognostic factors, and the extent towhich prognosis influences both selection fortreatment and treatment outcome will be ofparticular relevance. For example, in evaluationsof childhood vaccines there are few indicators ofprognosis that could be used to influenceallocation, so randomisation may not be necessary(although there are dangers that allocation inclusters could be confounded by exposure toinfectious disease, which means that in practicerandomisation is recommended). By contrast,when patient factors could have a strong influenceon allocation or where prognosis is strongly linkedto outcomes (such as in cancer treatment), thenrandomisation is likely to be extremely important.Other biases in non-randomisedstudiesNon-randomised studies are also susceptible toattrition, detection and performance bias. Forexample, attrition bias will occur if there are dropouts,detection bias if the assessment of outcomesis not standardised and blinded, and performancebias if there are errors and inconsistencies in theallocation, application and recording ofinterventions (Table 1). All of these biases can alsooccur in RCTs, but there is perhaps potential fortheir impact to be greater in non-randomisedstudies which are usually undertaken withoutprotocols specifying standardised interventions,outcome assessments and data recordingprocedures. A comprehensive assessment of thevalidity of both randomised and non-randomisedstudies requires the assessment of all dimensionsof potential bias.

Health Technology Assessment 2003; Vol. 7: No. 27Case-mix adjustment methodsIn the absence of information on factorsinfluencing allocation, the traditional solution toremoving selection bias in non-randomised studieshas been to attempt to control for knownprognostic factors, either by design and/or byanalysis. Alternative statistical techniques forremoving bias in non-randomised studies arebriefly described by D’Agostino and Kwan 1 andinclude matching, stratification, covarianceadjustment and the propensity score analysis(Box 2). However, all statistical techniques maketechnical assumptions (regression models typicallyassume that the relationship between theprognostic variable and the outcome is linear)and the degree to which they can adequatelyadjust for differences between groups isunclear.Moses 18 lists three factors necessary for successfuladjustment: (1) knowledge of which variables mustbe taken into account; (2) measuring thosevariables in each participant; and (3) using thosemeasurements appropriately to adjust thetreatment comparison. He further states that weare likely to fail on all three counts: 18“We often don’t understand the factors that causepeople’s disease to progress or not; even if we knewthose factors, we might find they had not beenmeasured. And if they were measured, the correct wayto use them in adjustment calls for a theoreticalunderstanding we seldom have.”BOX 2 Commonly used methods of statistical adjustmentStandardisationParticipants are analysed in groups (strata) which havesimilar characteristics, the overall effect being estimatedby averaging the effects seen in each of the groups.RegressionRelationships between prognostic factors and outcomeare estimated from the data in hand, and adjustmentscalculated for the difference in average values of theprognostic factor between the two groups. Linearregression (or covariance analysis) is used for continuousoutcomes, logistic regression for binary outcomes.Propensity scoresPropensity probabilities are calculated for each participantfrom the data set, estimating their chance of receivingtreatment according to their characteristics. Treatmenteffects are estimated either by comparing groups thathave similar propensity scores (using matching orstratification methods), or by calculating a regressionadjustment based on the difference in average propensityscore between the groups.Use of adjustment therefore assumes thatresearchers know which are the most importantprognostic factors and that these have beenappropriately measured. Further, it cannotaddress the problem of unknown orunmeasurable prognostic factors, which may playa particular role in confounding by indication.Full adjustment for confounding by indicationwould require information on prognostic factorsinfluencing both disease progression andtreatment choice. 14 Where there is an associationbetween prognosis and treatment choice, it wouldseem that correlates of prognosis can only warn ofa problem, not control for it. Furthermore, if thedegree to which the markers are correlated withprognosis is different in diseased and nondiseasedpersons then the magnitude (anddirection) of the resulting bias will beunpredictable in an overall analysis. 14Scope of this projectThis report contains the results of three systematicreviews and two empirical studies all relating tothe internal validity of non-randomised studies.The findings of these five studies will be ofimportance to researchers undertaking newstudies of healthcare evaluations and systematicreviews, and also healthcare professionals,consumers and policy makers looking to use theresults of randomised and non-randomised studiesto inform their decision-making.The first systematic review looks at existingevidence of bias in non-randomised studies,critically evaluating previous methodologicalstudies that have attempted to estimate andcharacterise differences in results between RCTsand non-randomised studies.Two further systematic reviews focus on the issueof quality assessment of non-randomised studies.The first identifies and evaluates tools that can beused to assess the quality of non-randomisedstudies. The second looks at ways that studyquality has been assessed and addressed insystematic reviews of healthcare interventions thathave included non-randomised studies.The two empirical investigations focus on the issueof selection bias in non-randomised studies. Thefirst investigates the size and behaviour ofselection bias in evaluations of two specific clinicalinterventions and the second assesses the degreeto which case-mix adjustment corrects forselection bias.5© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Health Technology Assessment 2003; Vol. 7: No. 27Chapter 2Aims and objectivesThis project has considered the methods andevidence-base for evaluating non-randomisedintervention studies. It is restricted to designs 3–6described in Chapter 1 (Box 1), and thus excludescase–control designs. The report first presents thefindings from reviews of current evidence andexisting tools for evaluating non-randomisedstudies, and the application of these tools insystematic reviews. It concludes with reports of twonew empirical investigations of expected biases andmethods for correcting for bias in non-randomisedstudies. The specific objectives addressed in eachof the chapters are presented below.Chapters 3–5 report the results of three separatereviews of the literature.Chapter 3 reviews the current empirical evidenceconcerning bias associated with non-randomisedstudies of healthcare interventions. In particular, itconsiders the evidence:●●●●that there are inconsistencies between the resultsof randomised and non-randomised studiesthat non-randomised studies may besystematically biasedthat the results of non-randomised studies maybe more variable than the results of RCTsthat case-mix adjustment reduces systematicbias and variability in non-randomised studies.Chapter 4 reviews existing tools for assessing thequality of non-randomised studies to:●●●describe the content, development and usabilityof quality assessment tools for non-randomisedstudiesappraise the degree to which the assessmenttools evaluate specified quality domainsidentify the tools most suitable for use insystematic reviews of non-randomised studies,and to identify their limitations.In Chapter 5, the use of quality assessment insystematic reviews is considered. The review aimsto:●evaluate the quality assessments made insystematic reviews that have included nonrandomisedstudies●investigate whether relationships between studyquality and study results have been observed.Chapters 6 and 7 report the results of newempirical investigations obtained through analysisof non-randomised studies generated byresampling participants from two large,multicentre randomised trials. The rationale andmethodology are described in detail in Chapter 6.The investigation reported in Chapter 6evaluates:●●the degree of systematic bias introduced by theuse of historical and concurrent controls in nonrandomisedstudies of healthcare interventionsand the impact that this has on studyconclusionsthe degree to which results of non-randomisedstudies using historical and concurrent controlsare more variable than those of randomisedcontrolled trials and the impact that this has onstudy conclusions.In Chapter 7, the investigation is extended toevaluate the ability of case-mix adjustmentmethods to correct selection bias introduced innon-randomised designs. Specifically it considers:●●●●whether case-mix adjustment methods cancompensate for systematic bias introducedthrough the use of historical and concurrentcontrolswhether case-mix adjustment methods cancompensate for increased variability introducedthrough the use of historical and concurrentcontrolswhether case-mix adjustment methods cancorrect for bias introduced through mechanismsakin to allocation by indicationwhether there are differences in theperformance of stratification, logistic regressionand propensity score methods for adjusting forselection bias.Chapter 8 presents an overview of the findingsfrom all sections of the project and makesrecommendations concerning the evaluation ofnon-randomised studies and further research thatshould be undertaken.7© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Health Technology Assessment 2003; Vol. 7: No. 27Chapter 3Review of empirical comparisons of the results ofrandomised and non-randomised studiesIntroductionEvidence about the importance of design featuresof RCTs has accumulated rapidly during recentyears. 19–21 This evidence has mainly been obtainedby a method of investigation that has been termedmeta-epidemiology, a powerful but simpletechnique of investigating variations in the resultsof RCTs of the same intervention according tofeatures of their study design. 22 The processinvolves first identifying substantial numbers ofsystematic reviews each containing RCTs both withand without the design feature of interest. Withineach review, results are compared between thetrials meeting and not meeting each designcriterion. These comparisons are then aggregatedacross the reviews in a grand overall meta-analysisto obtain an estimate of the systematic biasremoved by the design feature. For RCTs, therelative importance of proper randomisation,concealment of allocation and blinding have allbeen estimated using this technique. 20,21 Theresults have been shown to be consistent acrossclinical fields, 23 providing some evidence thatmeta-epidemiology may be a reliable investigativetechnique. The method has also been applied toinvestigate sources of bias in studies of diagnosticaccuracy, where participant selection, independenttesting and use of consistent reference standardshave been identified as being the most importantdesign features. 24The use of meta-epidemiology has also beenextended from the comparison of design featureswithin a particular study design to comparisonsbetween study designs. In two separate HTAreports, the results of RCTs have been comparedwith those from non-randomised evaluationsacross multiple interventions to estimate the biasremoved by randomisation. 25,26 However, themeta-epidemiology method may be inappropriatefor between design comparisons due to:●meta-confounding: the existence of otherdifferences between randomised and nonrandomisedstudies which could impact on theirfindings, and●unpredictability in the direction of effect:there possibly being no overall systematic biasbut biases acting unpredictably in differentdirections with varying magnitudes.An alternative methodology for empiricallyinvestigating differences between study designs isintroduced in Chapter 5. First, we review theevidence provided by meta-epidemiologicalcomparisons of randomised and non-randomisedstudies of the importance of randomisation per se,and discuss weaknesses in the use of metaepidemiologyto make such a comparison. Thecomparisons are considered in regard to fourparticular issues, namely whether there isempirical evidence of:●●●●inconsistencies in findings between RCTs andnon-randomised studiessystematic differences in average estimates oftreatment effects between RCTs and nonrandomisedstudiesdifferences in the variability of results betweenRCTs and non-randomised studies (betweenstudy heterogeneity), andwhether case-mix adjustment in nonrandomisedstudies reduces systematic biasand/or between study heterogeneity.MethodsReviews were eligible for inclusion if:●●they compared quantitative results betweenRCTs of an intervention and non-randomisedstudies of the same interventionthey had accumulated, through some systematicsearch, results from several of thesecomparisons across healthcare interventions.Reviews were identified from a search of electronicdatabases including MEDLINE, EMBASE andPsycLit, from the earliest possible date up toDecember 1999; from handsearches of Statistics inMedicine, Statistical Methods in Medical Research,Psychological Bulletin and Controlled Clinical Trials9© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Review of empirical comparisons of the results of randomised and non-randomised studies10and from contact with experts working in the field.The electronic search strategy used is provided inAppendix 1. Additionally, the searches carried outfor other sections of the project (includingsearches from the Cochrane Library databases)were screened to identify suitable papers forinclusion. Difficulties with devising searches formethodological topics are discussed in Chapter 4.The content, results and conclusions from each ofthe identified reviews were noted. In addition, themethodology of each review was critically assessedfor potential weaknesses. Aspects considered wereas follows:1. Was the identification of included studiesunlikely to be biased?2. Did the RCTs and non-randomised studiesrecruit similar participants, use similarinterventions and measure similar outcomes?3. Were the RCTs and non-randomised studiesshown to use similar study methodology in allrespects other than the allocation mechanism?4. Were sensible, objective criteria used todetermine differences or equivalence of studyfindings?ResultsEight reviews were identified which fulfilled theinclusion criteria; seven considered medicalinterventions and one psychological interventions.Brief descriptions of the methods and findings ofeach review are given below, with summary detailsgiven in Table 2. There is substantial overlap in theinterventions (and hence studies) that wereincluded in the reviews of medical interventions(Table 3).Description of the identified reviewsSacks, Chalmers and Smith 27Sacks and colleagues compared the results ofRCTs with historically controlled trials (HCTs).The studies were identified in Chalmers’ personalcollection of RCTs, HCTs and uncontrolled studiesmaintained since 1955 by searches of IndexMedicus, Current Contents and references of reviewsand papers in areas of particular medical interest(full list not stated). Six interventions were includedfor which at least two RCTs and two HCTs wereidentified [cirrhosis with oesophageal varices,coronary artery surgery, anticoagulants for acutemyocardial infarction, 5-fluorouracil adjuvanttherapy for colon cancer, bacille Calmette–Guérinvaccine (BCG) adjuvant immunotherapy anddiethylstilbestrol for habitual abortion (Table 3)].Trial results were classified as positive if there waseither a statistically significant benefit or if theauthors concluded benefit in the absence ofstatistical analysis, otherwise as negative. For eachof the six interventions, a higher percentage ofHCTs compared with RCTs concluded benefit:across all six interventions 20% of RCTs showedbenefit compared with 79% of the HCTs.The authors also considered whether statisticaladjustment for confounding factors in the HCTsreduced the discrepancy between the randomisedand non-randomised results. They found that theadjusted studies showed nearly the same treatmenteffect as the unadjusted studies.Kunz and Oxman 28 and Kunz, Vist and Oxman 29Kunz and Oxman searched the literature forreviews that made empirical comparisons betweenthe results of randomised and non-randomisedstudies. They included the results of the sixcomparisons in Sacks and colleagues’ study above,and results from a further five publishedcomparisons [antiarrhthymic therapy for atrialfibrillation, allogenic leucocyte immunotherapyfor recurrent miscarriage, contrast media forsalpingography, hormonal therapy forcryptorchidism, and transcutaneous electrical nervestimulation (TENS) for postoperative pain (Table 3)].In some of the comparisons, RCTs were comparedwith truly observational studies and, in others theywere compared with quasi-experimental trials. Aseparate publication of anticoagulants for acutemyocardial infarction already included in Sacksand colleagues’ review was also reviewed, 30 as was acomparison of differences in control group eventrates between randomised and non-randomisedstudies for treatments for six cancers (which doesnot fit within our inclusion criteria). 31 The reviewwas updated in 2002 including a further 11comparisons, and published as a Cochranemethodology review. 29The results of each empirical evaluation weredescribed, but no overall quantitative synthesis wascarried out. The results showed differencesbetween RCTs and non-randomised studies in 15of the 23 comparisons, but with inconsistency inthe direction and magnitude of the difference. Itwas noted that non-randomised studiesoverestimated more often than they underestimatedtreatment effects.Britton, McKee, Black, McPherson, Sandersonand Bain 25Britton and colleagues searched for primarypublications that made comparisons between

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.TABLE 2 Description of studies comparing results of randomised and non-randomised studiesNumber ofcomparisonsOther studydesignsincludedNumbers ofRCTs/otherstudiesSummary ofoverallquantitativeresultsSacks, 1982 27 Kunz, 1998, 28 Britton, 1998 25 MacLehose, Benson, 2000 32 Concato, 2000 33 Ioannidis, 2001 34 Lipsey, 1996, 352002 29 (figures 2000 26 2001 36in parenthesesrefer to originalpublication)6 23 (11) 18 14 (38 outcomes) 19 5 45 76Historicallycontrolled trials(HCTs)50/56 263 (122)/246 (152) 46/41 31/68 83/53 55/44 240/168 Not stated79% of HCTsfound the therapyto be better thanthe controlregimen,compared to 20%of RCTsQuasi-experiments,historicallycontrolled trials(HCTs), patientpreference trialsIn 15 (9) of 23 (11)comparisons effectswere larger in nonrandomisedstudies,4 (1) studies hadcomparable results,whilst 4 (2)reported smallereffectsQuasi-experiments,naturalexperiments, andprospectiveobservationalstudiesQuasi-experimentaland observationalstudiesSignificant In 14 of 35differences werefound in 11 of 18comparisons, butthere wereinconsistencies inthe direction of thedifferencescomparisons thediscrepancy in RRwas 50%.Discrepancies weresmaller in “fairer”comparisonsObservationalstudiesOnly in 2 of 19comparisons didthe point estimateof observationalstudies lie outsidethe confidenceinterval for theRCTsCase–control andcohort studiesFor the five clinicaltopics consideredthe average resultsof the observationalstudies wereremarkably similarto those of theRCTsQuasi-experiments,cohort and case–control studiesORs of RCTs andnon-RCTs are highlycorrelated(r = 0.75).Differences beyondchance were notedin 16% ofcomparisons: twofolddiscrepancies inORs occurred in33%Non-randomisedcomparativestudiesMean effect sizes(SD) of 0.46 (0.28)for RCTs and 0.41(0.36) for nonrandomisedstudies. Meandifference in effectsizes was 0.5, butvaried between–0.6 and +0.8 SDscontinuedHealth Technology Assessment 2003; Vol. 7: No. 2711

12TABLE 2 Description of studies comparing results of randomised and non-randomised studies (cont’d)Conclusionabout bias(fromabstract)ConclusionaboutvariabilityConclusionabout casemixadjustmentSacks, 1982 27 Kunz, 1998, 28 Britton, 1998 25 MacLehose, Benson, 2000 32 Concato, 2000 33 Ioannidis, 2001 34 Lipsey, 1996, 352002 29 2000 26 2001 36Biases in patientselection mayirretrievablyweight theoutcome of HCTsin favour of newtherapiesOn average nonrandomisedstudiestend to result inlarger estimates oftreatment. It is notpossible to predictthe magnitude oreven direction ofpossible selectionbiases andconsequentdistortions oftreatment effectsResults of RCTs andnon-randomisedstudies do notinevitably differ; itmay be possible tominimise anydifferences byensuring thatsubjects included ineach type of studyare comparableQuasi-experimentaland observationalstudy estimates ofeffectiveness maybe valid if importantconfounding factorsare controlled forThere is littleevidence thatestimates oftreatment effects inobservationalstudies are eitherconsistently largeror qualitativelydifferent from thoseobtained inrandomisedcontrolled trialsThe results ofwell-designedobservationalstudies do notsystematicallyoverestimate themagnitude oftreatment ascompared withthose inrandomisedcontrolled trialsNo comment No comment No comment No comment No comment Observationalstudies had lessvariability in pointestimates thanRCTs of the sametopicAdjustment ofoutcomes in HCTsfor prognosticfactors did notappreciably changethe resultsNo commentDespite goodcorrelationbetweenrandomised trialsand nonrandomisedstudies– in particular,prospective studies– discrepanciesbeyond chance dooccur anddifferences inestimatedmagnitude oftreatment effect arevery commonBetween studyvariability is smalleramong RCTs thanprospective nonrandomisedstudies(p = 0.03) and allnon-RCTsNon-randomiseddesigns may yielddifferent observedeffects relative torandomiseddesigns, but thedifference is almostas likely torepresent anupward as adownward bias(from text)(p = 0.07)The effect ofadjustment forbaseline differencesbetween groups innon-randomisedstudies isinconsistentNo comment No comment No comment No comment No commentReview of empirical comparisons of the results of randomised and non-randomised studiesRR, risk ratio.

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 3 Clinical topics investigated in each of the medical reviewsClinical topicSacks, 1982 27Kunz, 1998, 28 2002 29Britton, 1998 25MacLehose, 2000 26Concato, 2000 33Benson, 2000 32Ioannidis, 2001 341 Treatment of cirrhotic patients with oesophageal varicies ✓ ✓ ✓2 Surgical versus medical treatment of coronary artery disease ✓ ✓ ✓ ✓ ✓ ✓3 Anticoagulants for acute myocardial infarction ✓ ✓ ✓ ✓4 5-Fluorouracil adjuvant therapy for colon cancer ✓ ✓ ✓5 BCG adjuvant immunotherapy for malignant melanoma ✓ ✓ ✓6 Diethylstilbestrol for habitual abortion ✓ ✓ ✓7 Antiarrhythmic therapy for atrial fibrillation ✓ ✓ ✓ ✓8 Allogenic leucocyte immunotherapy for recurrent miscarriage ✓ ✓ ✓9 Oil soluble contrast media during hysterosalpingography for infertile couples ✓ ✓ ✓ ✓10 Hormonal therapy in cryptorchidism ✓ ✓ ✓11 TENS for postoperative pain ✓ ✓12 Matching of cancer control groups for six different cancers ✓13 β-Blockers for acute myocardial infarction ✓ ✓ ✓14 Surgical versus medical management of severe throat infections ✓ ✓15 Adenoidectomy for persistent otitis media ✓ ✓16 Chemotherapy, radiotherapy and tamoxifen in women with breast cancer ✓ ✓17 Treatment of benign oesophageal stricture ✓18 Day hospital versus inpatient care for alcoholism ✓19 Chorionic villus sampling versus early amniocentesis for karyotyping ✓ ✓20 Hospice versus conventional care for the terminally ill ✓21 Measles vaccine ✓22 Adjuvant doxorubin for treatment of sarcoma ✓23 Acute non-lymphatic leukaemia following adjuvant treatment for breast cancer ✓ ✓24 Antioxidant vitamins for cardiovascular protection ✓25 CABG vs PTCA for coronary artery disease ✓ ✓26 Calcium antagonists for acute coronary disease ✓ ✓27 Stroke units ✓28 Malaria vaccines ✓29 Breast conservation versus mastectomy for breast cancer ✓30 Tamoxifen versus placebo for breast cancer ✓31 Formula supplementation for infant feeding ✓32 Folic acid supplementation for prevention of neural tube defects ✓33 5-Fluorouracil adjuvant therapy for stomach cancer ✓34 Mammographic screening ✓ ✓ ✓35 BCG vaccine for prevention of tuberculosis ✓ ✓36 Treatment of cholesterol and death due to trauma ✓37 Treatment of hypertension and stroke ✓38 Treatment of hypertension and coronary heart disease ✓ ✓39 Intensive versus conventional insulin ✓40 Pneumatic retinopexy versus scleral buckling ✓41 Geriatric unit versus medical ward care (unclear for what condition) ✓42 Local versus general anaesthesia for endarcterectomy ✓ ✓43 Lithotripsy versus nephrolithotomy ✓44 Laser versus electrosurgical salpingostomy ✓45 Eder–Puestow versus balloon dilation ✓46 Hormone replacement therapy for osteoporosis ✓ a ✓47 Calcium channel blockers for renal graft survival ✓ a ✓48 Laparoscopic versus open appendectomy ✓ a ✓49 Naltrexone for alcohol dependence ✓50 Low-level laser therapy for osteoarthritis ✓51 Vaccine for meningitis serotype A ✓continued13© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Review of empirical comparisons of the results of randomised and non-randomised studiesTABLE 3 Clinical topics investigated in each of the medical reviews (cont’d)Clinical topicSacks, 1982 27Kunz, 1998, 28 2002 29Britton, 1998 25MacLehose, 2000 26Concato, 2000 33Benson, 2000 32Ioannidis, 2001 3452 Microsurgical vs macrosurgical salpingostomy in subfertility ✓53 Faecal occult blood screening for colorectal cancer ✓54 Intrathecal therapy for tetanus ✓55 Vitamin A supplementation ✓56 Interferon for hepatitis C ✓57 Trial of labour versus no labour for breech delivery ✓58 Surgical treatments for incontinence ✓59 Nonoxynol-9 spermicides for sexually transmitted diseases ✓60 Safety of postpartum discharge ✓61 High-dose diuretics as first-line treatment of hypertension ✓62 Aspirin for prevention of hypertension in pregnancy ✓63 Allogenic blood transfusion in cancer ✓64 Subcutaneous versus intravenous heparin in deep vein thrombosis ✓65 Low-protein diets in chronic renal insufficiency ✓66 Corticosteroids for idiopathic facial nerve palsy ✓67 Graduated compression stockings ✓68 Aspirin for primary prevention of stroke ✓69 Fixed nail plates versus sliding hip for femoral fractures ✓70 Corticosteroids for stable COPD ✓71 Treatment of hypertension in elderly people ✓72 Selective decontamination of the digestive tract ✓73 Cyclosporin withdrawal ✓74 Prehospital thrombolysis ✓75 T3-therapy in euthyroid depressed patients ✓ a76 Chemotherapy for oesophageal cancer ✓ a77 Autologous blood donation ✓ a78 Prevention of adolescent pregnancy ✓ a79 Educational interventions related to smoking and alcohol ✓ a80 Educational interventions related to nutrition and weight ✓ a81 Educational interventions related to other health behaviours ✓ a82 Medical and surgical management of coronary artery disease ✓ aaIndicates comparison added in review update.CABG, coronary artery bypass graft; PTCA, percutaneous transluminal coronary angioplasty; COPD, chronic obstructivepulmonary disease.14single randomised and non-randomised studies(14 comparisons) and secondary publications(reviews) making similar comparisons (fourcomparisons). Both observational and quasiexperimentalstudies were included in the nonrandomisedcategory. They included all four of thesecondary comparisons included in the review byKunz and colleagues 28 (Table 3). The single studycomparisons included studies where a comparisonwas made between participants who were allocatedto experimental treatment as part of a trial and agroup who declined to participate, and studies ofcentres where simultaneous randomised andpatient-preference studies had been undertaken ofthe same intervention. The studies were assessedto ensure that the randomised and nonrandomisedstudies were comparable on severaldimensions (Table 4).There were statistically significant differencesbetween randomised and non-randomised studiesfor 11 of the 18 comparisons. The direction ofthese differences was inconsistent and themagnitude extremely variable. For someinterventions the differences were very large.For example, in a review of treatments for acute

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.TABLE 4 Details of methodology of empirical comparisons between randomised and non-randomised studiesMethod ofidentificationofcomparisonsSacks, 1982 27 Kunz, 1998, 28 Britton, 1998 25 MacLehose, Benson, 2000 32 Concato, 2000 33 Ioannidis, 2001 34 Lipsey, 1996, 352002 29 2000 26 Wilson, 2001 36Primary studies:Personalsystematiccollection of RCTsand HCTs inspecific fields ofinterestSimilarity of Same interventionpatients, for the sameinterventions medical conditionandoutcomesSecondarystudies: Electronicand manual searchfor studiescomparingrandomised andnon-randomisedstudies2 studies controlledfor clinicaldifferences inparticipants andinterventions, 6 didso partly and 7 didnot at allPrimary and Primary andsecondary secondarystudies: Electronic studies: Electronicsearches for studies and manualcomparing searches for studiesrandomised and comparingnon-randomised randomised andgroupsnon-randomisedgroupsSame intervention,similar setting,similar controltherapy,comparableoutcomesmeasuresSame intervention.Similarity ofeligibility, timeperiod, comorbidity,diseaseseverity and otherprognostic factorsassessedPrimary studies:Electronic search ofMedline and CCTRfor observationalstudies andmatching RCTsRestricted tointerventionsallocated byphysicians to inducecomparabilitySecondarystudies: Metaanalysespublishedin 5 leading journalsthat included nonrandomisedstudiesSecondarystudies: Metaanalyseslocated byelectronic searches(MEDLINE andCochrane library)and manualsearchesSecondarystudies: Metaanalyseslocated byelectronic searches(Psychology andSociologydatabases) andmanual searchesNot assessed Not assessed Impact ofdifferencesconsidered inmultivariateanalysisSimilarity ofother studymethodsNot assessed5 papers judgedpartly comparable,10 not comparableon double blinding,complete follow-upand othermethodologicalissuesNot assessedBlinding of outcomeassessedNot assessedAnalysis accordingto study qualitymentioned indiscussion, but nomethods or resultspresentedNot assessedImpact ofassessments ofstudy qualityconsidered inmultivariateanalysiscontinuedHealth Technology Assessment 2003; Vol. 7: No. 2715

16TABLE 4 Details of methodology of empirical comparisons between randomised and non-randomised studies (cont’d)Method ofsummarisingstudyfindingsMethod forcomparingresultsbetweengroupsCriteria forcomparingvariability ofresultsSacks, 1982 27 Kunz, 1998, 28 Britton, 1998 25 MacLehose, Benson, 2000 32 Concato, 2000 33 Ioannidis, 2001 34 Lipsey, 1996, 352002 29 2000 26 2001 36Vote counting ofresults classified aspositive (eitherstatisticallysignificant or hadpositiveconclusions if nostatistical analysis)Comparison of thepercentage withpositive resultsNo consistentsummary: results ofrandomised andnon-randomisedgroups describedusing a variety ofmeasuresNo overall analysispresentedResults ofrandomised andnon-randomisedgroups describedusing riskdifferences, RRs orORsStatisticalsignificance of thedifference in effectsizesRRs and riskdifferencescalculatedseparately for MAsfor randomised andnon-randomisedstudiesDistribution ofrelative andabsolute differencesin results reportedCalculation ofoverall MA resultsand CIs (ORs forbinary data,differences inmeans forcontinuous)Assessment ofwhetherobservational pointestimate fell within95% CI for RCTsCalculation ofoverall MA resultsand CIs (RRs andORs)No overall analysispresentedNot assessed Not assessed Not assessed Not assessed Not assessed Dispersion of pointscalculated withoutconsideringdifferences insample sizeCalculation of fixedand random MAestimates expressedas ORs andlog ORsCalculation ofZ-scores fordifference betweentreatment effectsSignificance(p < 0.1) of tests ofbetween studyheterogeneity(1) Mean effectsizes of allrandomised and allnon-randomisedstudies.(2) distribution ofdifferences ineffect sizesbetweenrandomised andnon-randomised ineach MAGraph ofdistribution ofdifferences ofeffect sizesNot assessed,although standarddeviations ofrandomised andnon-randomisedstudies presented(unadjusted fordifferences insample size)Review of empirical comparisons of the results of randomised and non-randomised studiesCCTR, Cochrane Controlled Trials Register; MA, meta-analysis.

Health Technology Assessment 2003; Vol. 7: No. 27non-lymphatic leukaemia, the risk ratio in RCTswas 24 compared with 3.7 in non-randomisedstudies (comparison 23 in Table 3).The impact of statistical adjustment for baselineimbalances in prognostic factors was investigatedin two primary studies, and in four additionalcomparisons (coronary angioplasty versus bypassgrafting, calcium antagonists for cardiovasculardisease, malaria vaccines and stroke unit care:comparisons 25–28 in Table 3). In two of the sixcomparisons there was evidence that adjustmentfor prognostic factors led to improvedconcordance of results between randomised andnon-randomised studies.MacLehose, Reeves, Harvey, Sheldon, Russelland Black 26MacLehose and colleagues restricted their reviewto studies where results of randomised and nonrandomisedcomparisons were reported togetherin a single paper, arguing that such comparisonsare more likely to be of ‘like-with-like’ than thosemade between studies reported in separate papers.They included primary studies and also reviewsthat pooled results from several individual studies.Of the 14 comparisons included in their report,three were based on reviews (comparisons 3, 7 and25 in Table 3) and the rest were results fromcomparisons within single studies. The nonrandomiseddesigns included comprehensivecohort studies, other observational studies andquasi-experimental designs.The ‘fairness’ or ‘quality’ of each of the comparisonsmade was assessed for comparability of patients,interventions and outcomes and additional studymethodology (see Table 4). Although the authorsdid not categorise comparisons as showingequivalence or discrepancy, the differences inresults were found to be significantly greater incomparisons ranked as being low quality.Benson and Hartz 32Benson and Hartz evaluated 19 treatmentcomparisons (eight in common with Britton andcolleagues 25 ) for which they located at least onerandomised and one observational study (definedas being a study where the treatment was notallocated for the purpose of research) in a search ofMEDLINE and the databases in the CochraneLibrary (Table 4). They only considered treatmentsadministered by physicians. Across the 19comparisons they found 53 observational and 83randomised studies, the results of which were metaanalysedseparately for each treatment comparison.Comparisons were made between the pooledestimates, noting whether the point estimate fromthe combined observational studies was within theconfidence interval of the RCTs. They found onlytwo instances where the observational andrandomised studies did not meet this criterion.Concato, Shah and Horwitz 33Concato and colleagues searched for meta-analysesof RCTs and of observational studies (restricted tocase–control and concurrent cohort studies)published in five leading general medical journals.They found only five comparisons where bothtypes of study had been meta-analysed [BCGvaccination for tuberculosis (TB), mammographicscreening for breast cancer mortality, cholesterollevels and death from trauma, treatment ofhypertension and stroke, treatment of hypertensionand coronary heart disease (CHD) (Table 3)]combining a total of 55 randomised and 44observational studies. They tabulated the results ofmeta-analyses of the randomised and theobservational studies and considered the similarityof the point estimates and the range of findingsfrom the individual studies. In all five instancesthey noted the pooled results of randomised andnon-randomised studies to be similar. Whereindividual study results were available, the range ofthe RCT results was greater than the range of theobservational results.Ioannidis, Haidich, Pappa, Pantazis, Kokori,Tektonidou, Contopoulous-Ioannidis and Lau 34Ioannidis and colleagues searched for reviews thatconsidered results of RCTs and non-randomisedstudies. In addition to searching MEDLINE theyincluded systematic reviews published in theCochrane Library, locating in total 45comparisons. Comparisons of RCTs with bothquasi-randomised and observational studies wereincluded. All meta-analytical results wereexpressed as odds ratios, and differences betweenrandomised and non-randomised resultsexpressed as a ratio of odds ratios and theirstatistical significance calculated. Findings acrossthe 45 topic areas were pooled incorporatingresults from 240 RCTs and 168 non-randomisedstudies. Larger treatment effects were noted moreoften in non-randomised studies. In 15 cases(33%) there was at least a twofold variation in oddsratios, whereas in 16% there were statisticallysignificant differences between the results ofrandomised and non-randomised studies. Theauthors also tested the heterogeneity of the resultsof the randomised and non-randomised studiesfor each topic. Significant heterogeneity was notedfor 23% of the reviews of RCTs and for 41% of thereviews of non-randomised studies.17© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Review of empirical comparisons of the results of randomised and non-randomised studies18Lipsey and Wilson 35 and Wilson and Lipsey 36Lipsey and Wilson searched for all meta-analysesof psychological interventions, broadly defined astreatments whose intention was to inducepsychological change (whether emotional,attitudinal, cognitive or behavioural). Evaluationsof individual components of interventions andbroad interventional policies or organisationalarrangements were excluded. Searches ofpsychology and sociology databases supported bymanual searches identified a total of 302 metaanalyses,76 of which contained both randomisedand non-randomised comparative studies. Resultswere analysed in two ways. First, the average effectsizes of randomised and non-randomised studieswere computed across the 74 reviews, and averageeffects were noted to be very slightly smaller fornon-randomised than randomised studies. Second(and more usefully) the difference in effect sizesbetween randomised and non-randomised studieswithin each of the reviews was computed andplotted. This revealed both large over- andunderestimates with non-randomised studies,differences in effect sizes ranging from –0.60 to+0.77 standard deviations.Studies excluded from the reviewThree commonly cited studies were excluded fromour review. 37–39 Although these studies madecomparisons between the results of randomisedand non-randomised studies across manyinterventions, they did not match RCTs and nonrandomisedstudies according to the intervention.Although they provide some information about theaverage findings of selected randomised and nonrandomisedstudies, they did not consider whetherthere are differences in results of RCTs and nonrandomisedstudies of the same intervention.Findings of the eight reviewsThe eight reviews have drawn conflictingconclusions. Five of the eight reviews concludedthat there are differences between the results ofrandomised and non-randomised studies in manybut not all clinical areas, but without there being aconsistent pattern indicating systematicbias. 25,26,28,34,35 One of the eight reviews found anoverestimation of effects in all areas studied. 27 Thefinal two concluded that the results of randomisedand non-randomised studies were ‘remarkablysimilar’. 32,33Of the two reviews that considered the relativevariability of randomised and non-randomisedresults, one concluded that RCTs were moreconsistent 34 and the other that they were lessconsistent. 33The two studies that investigated the impact ofcase-mix adjustment were in agreement, bothnoting that adjustment did not necessarily reducediscordance between randomised and nonrandomisedfindings. 25,27Critical evaluation of reviewsThe discrepancies in the conclusions of the eightreviews may in part be explained by variations intheir methods and rigour, so that they had varyingsusceptibility to bias. We consider the weaknessesin these reviews under four headings.Was the identification of included studiesunlikely to be biased?The studies used in all the reviews represent onlya very small portion of all randomised andobservational research. From Table 3, it is clear thatthe seven reviews in medical areas were each onlybased on a subset of known comparisons ofrandomised and non-randomised evidence. Eventhe largest review 34 did not include allcomparisons identified in previous reviews.Correspondence has also cited several otherexamples of treatment comparisons where thereare disagreements between observational andrandomised studies. 40,41More important, could the comparisons selectedfor these meta-epidemiological reviews be apotentially biased sample? There are two levels atwhich publication bias can act in these evaluations:(a) selective publication of primary studies, whichwill affect all reviews, and (b) selective publicationof meta-analyses of these studies, which will affectreviews restricted to secondary publications.Evaluations of publication bias have noteddifferences in the frequency of primary publicationof randomised and observational studies, althoughthe direction and magnitude of the differencesvary between evaluations and the relationship tostatistical significance is not known. 42 Similarly,the decisions made concerning the publication ofmeta-analyses that include non-randomisedstudies are likely to be influenced by the results ofexisting randomised controlled trials. TheConcato review 33 may be the most susceptible topublication bias as it restricted study selection tometa-analyses combining randomised orobservational results published in five generalmedical journals: Annals of Internal Medicine, BritishMedical Journal (BMJ), Journal of the AmericanMedical Association (JAMA), New England Journal ofMedicine (NEJM) and The Lancet. Therefore, theonly studies eligible for inclusion were those whereboth authors and top journal editors had alreadydecided that it was sensible to synthesise the

Health Technology Assessment 2003; Vol. 7: No. 27results of both observational and randomisedstudies. It seems highly likely that these decisionsmay relate to the similarity of the results of studiesof different designs.Did the RCTs and non-randomised studiesrecruit similar participants, use similarinterventions and measure similar outcomes?Discrepancies between the results of observationaland randomised studies may be confounded bydifferences in the selection and evaluation ofpatient groups, in the formulation and delivery oftreatments, in the use of placebos and othercomparative treatments and in the methods usedto maintain follow-up and record outcomes. Formany interventions there may also be temporalconfounding of study types, non-randomisedstudies typically being performed prior to theRCTs. Such meta-confounding will make it difficultto attribute a systematic difference directly to theuse or non-use of a random allocation mechanism.Six of the eight reviews noted this problem andincorporated features in their evaluations toreduce the potential for metaconfounding.25–28,32,35 Two reviews made morestringent efforts to assess comparability than theothers. 25,26 Britton and colleagues restricted theselection of studies to be similar in terms ofintervention, setting, control therapy and outcomemeasure. MacLehose and colleagues assessed eachcomparison for the possibility of metaconfoundingand found that the most susceptible(i.e. those with differences in eligibility criteria andtime periods and no adjustment for severity ofdisease, comorbidity and other prognostic factors)had, on average, the largest discrepancies.Were the RCTs and non-randomised studiesshown to use similar study methodology in allrespects other than the allocation mechanism?Meta-confounding could also occur throughdifferences in other aspects of study design,beyond the use of randomisation. For example,the results of RCTs are known to vary according tothe quality of randomisation, especially theconcealment of allocation at recruitment. 23However, none of the reviews restricted theinclusion of RCTs to those with adequateconcealment or on any other methodologicalbasis. Only one review 26 assessed the comparabilityof randomised and non-randomised studies onany aspect of study quality (blinding).Discrepancies and similarities between studydesigns could be partly explained by differences inother unevaluated aspects of the methodologicalquality of the RCTs.Similarly, there will be differences in themethodological rigour of the non-randomisedstudies. Importantly, the possible biases of nonrandomisedstudies vary with study design. Onlyone review 27 restricted non-randomised studies tobe of a single design (historically controlledstudies). In all the others, RCTs were comparedwith non-randomised studies of a mixture ofdesigns.Were sensible, objective criteria used todetermine differences or equivalence of studyfindings?The manner in which results were judged to be‘equivalent’ or ‘discrepant’ varied widely betweenthe reviews and influenced the conclusions thatwere drawn. For example, Concato andcolleagues 33 deemed that the randomised andnon-randomised studies of mammographicscreening (comparison 34 in Table 3) hadremarkably similar results, whereas Ioannidis andcolleagues 34 classified them as discrepant. Only intwo reviews was the judgement made aggregatedacross the comparisons. 27,35 In all the otherreviews each individual topic was classified aseither equivalent or discrepant. Many of thecomparisons made at the level of a clinical topicwere based on very few data, for example, in theIoannidis review 34 on average for eachintervention five RCTs were compared with fournon-randomised studies. Hence the absence of astatistically significant difference cannot beinterpreted as evidence of ‘equivalency’ andclinically significant differences in treatmenteffects cannot be excluded. Conversely, thepresence of a statistically significant differencedoes not indicate that a clinically importantdifference does exist. Four reviews 26,28,34,35 moreusefully concentrated on describing the magnitudeof the differences, all four noting substantialdifferences occurring in some, but not all,comparisons. The Concato review 33 subjectivelyclassified all comparisons as being ‘remarkablysimilar’.The comparisons made in two reviews 33,34 of therelative variability of randomised and nonrandomisedresults can be considered flawedowing to the criteria used to compare variation.Concato and colleagues considered the range ofthe point estimates from observational studies andRCTs. 33 This comparison was confounded by thedifferent sample sizes used in observational andrandomised studies, and the number of studiesconsidered. On average, the RCTs in the Concatoreview were 25% smaller than the observationalstudies, hence greater variability in their results is19© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Review of empirical comparisons of the results of randomised and non-randomised studies20to be expected. It is also possible that the mostextreme RCT results arose in particularly smallrandomised studies, and that there is an absenceof equivalently small observational studies.Ioannidis and colleagues investigated thestatistical significance of the heterogeneity. 34 Thisanalysis is also dependent on the sample size andnumber of studies considered, hence thecomparison between randomised and nonrandomisedstudies may be confounded.DiscussionThe conclusions of the eight reviews are divergent,and as all the reviews have weaknesses, it isdifficult to draw conclusions concerning theimportance of randomisation from theseinvestigations. The only robust conclusion that canbe drawn is that in some circumstances the resultsof randomised and non-randomised studies differ,but it cannot be proved that differences are notdue to other confounding factors. The frequency,size and direction of the biases cannot be judgedreliably from the information presented.One review reported the existence of a systematicbias, namely that historically controlled trials weremore likely to conclude benefit of new therapiesthan RCTs. 27 However, as the conclusions werebased on statistical significance and not size ofeffect, this may be confounded by sample size.Direct comparison of particular non-randomiseddesigns with RCTs was not made in any of theadditional seven reviews.Two reviews raised concerns about the usefulnessof case-mix adjustment methods in nonrandomisedstudies. 25,27 Again, the validity ofthese conclusions is uncertain owing to thepossibility of meta-confounding.The efforts used to control confounding variedbetween the reviews. The potential forconfounding in these comparisons requiresdetailed, clinically informed assessment of thesimilarity of populations, interventions andoutcome assessments used in the RCTs and nonrandomisedstudies. Six of the eight reviews arejudged to be of poor quality in this regard, as theymade no or little assessment of comparability. Thereviews by Britton and colleagues 25 and MacLehoseand colleagues 26 used the most comprehensiveassessments to judge comparability, especially forthe six comparisons they discussed in detail. Fortwo of the four interventions considered in Brittonand colleagues’ report, the only conclusion thatcould be drawn was that due to differences in era,populations, dosage and length of follow-up,differences between randomised and nonrandomisedstudies were to be expected. Brittonand colleagues implied that for these clinicalinterventions a ‘fair’ comparison of randomisedand non-randomised studies would not bepossible. MacLehose and colleagues concludedtheir review by recommending that onlycomprehensive cohort studies could make faircomparisons between randomised and nonrandomisedevidence. Few such studies have everbeen undertaken.These investigations also raise concerns regardingthe usefulness of meta-epidemiologicalinvestigations where there is (or could be)variability in the direction of bias. The metaepidemiologicalmethod is designed to identifyand estimate only systematic bias. Only if theselection of groups in non-randomised studiesconsistently led to over-representation of high- (orlow-) risk participants in one group over anotherwould the bias always be seen to act in the samedirection. If selection bias in non-randomisedstudies arises as a result of haphazard variations incase-mix, there will be a mixture of under- andoverestimates of the treatment effect. The resultsmight all be biased, but not all in the samedirection. In these circumstances, the differencebetween randomised and non-randomised studieswould manifest as an increase in variance orheterogeneity of treatment effect (beyond thatexpected by chance) rather than (or in additionto) a systematic bias. This possibility is perhapshinted at by the observation in five of the eightreviews that the differences observed betweenrandomised and non-randomised studies did notact in a consistent manner. Ideally, a formalstatistical comparison should aim to compare theheterogeneity in treatment effects, and not just theaverage treatment effects between randomised andnon-randomised groups. None of the eightcomparisons used a statistical model to compareresults and, in any case, even the random effectsmethod presently recommended for metaepidemiologicalstudies would be inadequate as itassumes that the variance of effects in all groupsare equal. 22These major problems are in contrast to thesuccessful application of the meta-epidemiologicalmethod to understand the importance ofparticular design features for RCTs. 20,21,23,43 Thepotential for meta-confounding in RCTs may bereduced if the types of participants, interventionsand outcomes are more similar across trials than

Health Technology Assessment 2003; Vol. 7: No. 27across non-randomised studies, and there arefewer methodological characteristics of interest(randomisation, concealment of allocation, blindingand completeness of follow-up). Also, there areclear reasons to believe that the biases againstwhich these design features protect are likely to actin favour of a particular treatment (theexperimental treatment), leading to systematic bias,such that the problem of increased heterogeneitybetween study designs does not arise.To take account of our concerns about thecomplexity of biases in non-randomised studies,we have first undertaken a review of qualitydomains and assessment methods used to considerthe likelihood of bias in non-randomised studies(Chapters 4 and 5). Second, we have undertakennovel methodological assessments of theimportance of two quality issues which are specificto non-randomised studies: method of allocation(Chapter 6) and use of case-mix adjustment(Chapter 7). Our methodological assessments havebeen designed to overcome the particularproblems of meta-confounding and variability ofdirection and size of bias that plague theinterpretation of these previous reviews.21© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Health Technology Assessment 2003; Vol. 7: No. 27Chapter 4Evaluation of checklists and scales for assessingquality of non-randomised studiesIntroductionAs discussed in Chapter 1, the RCT is widelyregarded as the design of choice for theassessment of the effectiveness of healthcareinterventions as it is theoretically most likely toproduce an unbiased estimate of effect (has highinternal validity). Nevertheless, there is anincreasing recognition of the role of nonrandomisedstudies for the assessment ofeffectiveness, either to support or generalise theresults of RCTs or because RCTs simply do notexist and/or are not possible to conduct.Regardless of the study designs available, thevalidity of any estimate of effectiveness isconditional on the quality of the study or studiesupon which that estimate is based. The quality of astudy has been defined as “the confidence that thetrial design, conduct and analysis has minimisedor avoided biases in its treatment comparisons”. 44This definition places the focus on questions ofinternal validity, and has been adopted for use inthis report. Although many would argue that themain benefit from non-randomised studies lies inincreasing the external validity of study findings, ifwe cannot establish the internal validity of nonrandomisedstudies then the external validity ofthe results becomes largely irrelevant.Study quality is a rather subjective concept, opento different interpretations depending on thereader. Regular readers of healthcare literaturedevelop informal approaches to assessing quality,based on information picked up from a variety ofsources, including previous experience anddiscussions with others. Where a study, orsystematic review of studies, is to be used toevaluate the effectiveness of an intervention, moreformal approaches to assessing quality arerequired. As Cooper 45 has pointed out, “almostevery primary researcher and research reviewerbegins an inquiry with some idea about theoutcome of the enquiry”, and these“predispositions toward a review’s results caninfluence the reviewers’ judgements about themethodological quality of a piece of research”.Even short and apparently simple tools, such asthe scale for assessing the quality of RCTsdeveloped by Jadad and colleagues, 46 havebeen found to have low inter-rater reliability. 47Further examples are cited by Cooper, 45 whoconcludes that there are two main sources ofvariance in evaluators’ decisions: the relativeimportance that they assign to different researchcharacteristics and their judgements abouthow well a particular study meets a designcriterion.A formal approach to quality assessment intuitivelyseems the best way of reducing the level ofsubjectivity that can be introduced. Approaches toquality assessment largely follow two mainframeworks. 45,48 The first approach follows workin the social sciences field largely by Campbell andcolleagues, 49,50 who outlined the various ‘threats tovalidity’ that can be encountered in experimentaland quasi-experimental studies. This workoriginally focused on internal and externalvalidity, but evolved over the years to cover‘statistical conclusion validity’ and ‘constructvalidity’. Within each category, a list of specificthreats to validity was provided; the final versionlists 33 threats to validity across the fourcategories. 50 Although perhaps not originallyintended to be used as such, this work has beenused as a basis for quality assessment. One ormore of the threats to validity may be selected andreviewers assess whether or not they are present inany given study. The advantage of the approach isthat relevant validity threats can be selected froman explicit list, which then contribute to an overalljudgement of quality.The second approach is the ‘methods-description’approach, whereby the reviewer codes the objectivecharacteristics of each study’s methods as they aredescribed by the primary researchers. 45 Among thefirst proponents of this approach were Chalmersand colleagues, 51 who outlined a detailed list ofcriteria with which to assess (and score) the qualityof RCTs. The objective of this method is to providean overall index of quality rather than to estimatethe degree of bias present. 48 As with the threats to23© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Evaluation of checklists and scales for assessing quality of non-randomised studies24validity approach, different reviewers may chooseto list different methodological characteristics.However, one of the main advantages is that thecoding of study characteristics does not require thesame degree of judgement as when one is requiredto identify the presence of a threat to validity. Forexample, two reviewers might disagree on whetheror not a study is low in power (threat to validity),and yet have perfect agreement when coding theseparate components that make up the decision(e.g. sample size, study design, inherent power ofthe statistical test). 45Cooper 45 advocates that the optimal strategy forcategorising studies is a mix of these twoapproaches, that is, coding all potentially relevant,objective aspects of research design as well asspecific threats to validity which may not be fullycaptured by the first approach. Although thisstrategy does not (and could not) remove all of thesubjectivity from the assessment process, it may bethe best way of making assessments of quality asexplicit and objective as possible. It should benoted that inter-rater reliability is likely to befurther diminished where a judgement regardingthe ‘acceptability’ of a given feature is required asopposed to identifying its presence or absence. 52Quality assessment tools developed for thehealthcare literature have followed bothapproaches, but the majority are of the ‘methodsdescription’variety, whether they took the form ofa checklist or a scale. These tools provide a meansof judging the overall quality of a study usingitemised criteria, either qualitatively in the case ofchecklists or quantitatively for scales. 53Alternatively, a component approach can be taken,whereby one or more individual qualitycomponents, such as allocation concealment orblinding, are investigated. However, as Moher andcolleagues have pointed out, “assessing onecomponent of a trial report may provide onlyminimal information about its overall quality”. 53A common criticism of quality assessment tools isthe lack of rationale provided for the particularstudy features that reviewers choose to code 52 andthe inclusion of features unlikely to be related tostudy quality. 44,54 This may in part be due to thelack of empirical evidence for the biases associatedwith inadequately designed studies (although suchevidence does exist to some extent forRCTs 21,55,56 ). A further criticism of tools forassessing RCTs is lack of attention to standardscale development techniques, 44 to the extent thatone scale 46 which was developed usingpsychometric principles has been singled out fromother available tools. These principles involve thefollowing steps as laid out by Streiner andNorman: 57 preliminary conceptual decisions; itemgeneration and assessment of face validity; fieldtrials to assess frequency of endorsement,consistency and construct validity; and generationof a refined instrument. However, as Jüni andcolleagues have pointed out, following suchprinciples does not necessarily make a toolsuperior to other available instruments. 23Quality assessment scales in particular have alsobeen heavily criticised for the use of a singlesummary score to estimate study quality, by addingthe scores for each individual item. 58 Greenlandargues that the practice of quality scoring is themost insidious form of bias in meta-analysis as it“subjectively merges objective information witharbitrary judgements in a manner that can obscureimportant sources of heterogeneity among studyresults”. 58 It has since been empiricallydemonstrated that the use of different qualityscales for the assessment of the same trial(s) resultsin different estimates of quality. 21,59 Nevertheless,formal (or systematic) quality assessment,especially of RCTs, is increasingly common. Areview by Moher and colleagues published in 1995identified 25 scales for the quality assessment ofRCTs. 44 Subsequent work by Jüni and colleagueshas identified several more. 23,59In spite of these criticisms, it is largely agreed thatthe assessment of methodological quality shouldbe routine practice in systematic reviews and metaanalyses.Although the majority of methodologicalwork in this area has surrounded the assessment ofRCTs, it is reasonable to suggest that if formalquality assessment of randomised controlled trialsis important, then it is doubly so for nonrandomisedstudies owing to the greater degree ofjudgement that is required. The largelyobservational nature of non-randomised studiesleads to a much higher susceptibility to bias thanis found for experimental designs, as discussed inChapter 1.A review of existing quality assessment tools for nonrandomisedintervention studies was conducted inorder to provide a description of what is available,paying particular attention to whether and how wellthey cover generally accepted quality domains.MethodsInclusion criteriaTo be considered as a quality assessment tool, a list

Health Technology Assessment 2003; Vol. 7: No. 27of criteria that could be (or had been) used toassess the methodological quality or validity ofprimary studies was required. A specific statementthat the list of criteria was a scale or checklistintended to assess methodological quality was notrequired.These tools could exist either as individualpublications in their own right or within thecontext of a systematic review or other type ofreview, such as methodological reviews that hadused some form of tool.The tool must have been (or must have thepotential to be) applied to non-randomisedstudies of intended effect, that is, epidemiologicalstudies or studies primarily aimed at theinvestigation of the side-effects of an interventionwere excluded. Tools specifically designed to assesscase–control and uncontrolled studies wereexcluded (case–control studies were excluded onthe basis that the design is rarely used to examineintended effects). To provide as comprehensive apicture of current practice as possible, any toolthat had been used to assess non-randomisedstudies was included in the review, even if it hadbeen explicitly designed to assess only RCTs.Both ‘new’ and ‘modified’ tools were included.‘Modified’ tools were those based on a singleexisting tool. Tools that were stated to be based onmore than one existing tool were considered to be‘new’ tools. Tools which made no statementregarding the originality of the tool were assumedto be ‘new’ tools. This is likely to have led to anover-estimation of the number of unique tools inexistence; however, it was not practical to checkfurther on the origin of these tools.Literature searchesIn an attempt to identify the largest possiblenumber of quality assessment tools, an extensiveand comprehensive literature search from theearliest possible date up to December 1999 wascarried out. This included searching a wide rangeof electronic databases (see Table 5). The searchstrategies used were developed via an iterativeprocess, by which a series of strategies weresuggested, amended and piloted, and the searchresults scanned to identify the proportion ofrelevant papers retrieved (see Appendix 1 for thesample search strategy). Owing to the nature ofthe searches, and the poor indexing of the studies,it was necessary to strike a balance betweenstrategies that were less likely to miss any relevantpapers, yet retrieved a ‘manageable’ number ofcitations. Similar problems with searching formethodological literature have been cited byprevious HTA-funded projects. 60 The resultinglists of titles and abstracts were screened by twoTABLE 5 Literature search resultsSource Retrieved Selected from Met inclusionscreening a criteriaMEDLINE (1966–99) 1897 149 26EMBASE (1974–99) 639 11 0PsycLit (1967–99) 1835 113 4Science Citation Index (1981–99) 1078 45 5Social Science Citation Index (1981–99) 502 11 0Index to Scientific and Technical Proceedings (1990–9) 294 6 0Applied Social Sciences Index and Abstracts (1987–99) 262 10 1Educational Resource Information Centre database (ERIC) (1965–99) 699 29 3British Education Index (1986–99) 11 0 0Cochrane Review Groups 3Citation searches NA NA 85Database of Abstracts of Reviews of Effectiveness (DARE) (1994–9) 1109 131 75 bOther c NA NA 11Total 8326 213a Totals presented are those following de-duplication of search results, i.e. only the additional number of unique studiesobtained from each source is presented.b Number of included reviews that developed their own quality assessment tool or modified another.c Includes the CRD and Cochrane Collaboration methodology databases, handsearching of a number of key journals(Statistics in Medicine (1984–98), Controlled Clinical Trials (1984–98), Journal of Clinical Epidemiology (1991–8), PsychologicalBulletin (1994–9), Psychological Methods (1996–9) and the International Journal of Technology Assessment in Health Care(1985–99)) and contact with a number of methodological experts.25© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Evaluation of checklists and scales for assessing quality of non-randomised studies26reviewers. The full papers for all records thatpotentially met our inclusion criteria wereobtained.Searches of primary electronic databases weresupplemented with searches of registers ofmethodological research, citation searches for keypapers, handsearching of key journals and contactwith experts. The reference lists of all retrievedpapers were also scanned to identify anyadditional tools.Data extractionA data extraction form for recording relevantinformation from each quality assessment tool wasdesigned and piloted (see Appendix 2). Data wereextracted by one reviewer and the completed dataextraction form was double-checked against theoriginal paper by a second reviewer. Anydisagreements were resolved by consensus or byreferral to a third reviewer where necessary.For each tool, items relating to the following areaswere recorded:1. Descriptive information. The purpose forwhich the tool had been developed (Box 3);whether it was a new tool or a modification ofan existing one; the type of study designscovered; the inclusion of design-specificquestions; whether it was a scale or a checklist;for scales only, the weighting system adopted;the number of items (both generic and topicspecific);and whether the tool had been usedin our identified systematic reviews.2. Tool development. Information relating to howthe items were generated; whether the selectionof items was justified; and whether and how thevalidity and/or reliability of the tool wasexamined.3. Tool content. To facilitate extraction of thecontent of the tools and allow comparisonbetween tools, a taxonomy of 12 qualityBOX 3 Purposes for which a tool could be developed1. Planning a study2. Assessing a statistical/methodological analysis3. Evaluating a study when considering the practicalapplication of the intervention4. Evaluating/grading study recommendations whenproducing recommendations/guidelines5. Peer reviewing6. Reporting a study7. Assessing studies included in a systematic reviewdomains covering the major aspects of studyquality was constructed a priori (Table 6). Thesedomains covered internal validity, externalvalidity and issues related to the quality ofreporting. A thirteenth domain was used forancillary issues not covered elsewhere. Withineach domain, items that might be expected toappear in a quality assessment tool werespecified. These domains and items weregenerated using a modified Delphi processamongst review team members. An initial list ofdomains and items generally known or believedto be important for the assessment of RCTs wasdrawn up. Additional items potentially relevantto the assessment of non-randomised studieswere then added.Six of the 12 domains are mainly related tointernal validity. Of these, we consider that the twomost important for the evaluation of nonrandomisedintervention studies are the creationof the intervention groups (domain #5) and thecomparability of the groups at the analysis stage(domain #9). For studies that are not randomised,it is important to know first how the allocationswere performed (for example, according toclinician or patient preference) and secondwhether attempts were made to ensure that thegroups were similar for key prognosticcharacteristics by design (such as through usingmatching). Furthermore, the attempts to identifyall important prognostic factors and the use ofcase-mix adjustment to account for any differencesbetween groups are thought to be particularlyimportant in non-randomised studies. These twodomains were considered core domains for theassessment of non-randomised studies, and withinthem four core criteria were specified (Table 6).Other domains related to internal validityincluded blinding (of patients and investigators)(domain #6); the soundness of information aboutthe intervention and the outcomes (domain #7);the adequacy of follow-up (domain #8); and theappropriateness of the analysis (domain #10). Thesoundness of information refers to the confidenceone has that the patients actually received theintervention to which they were assigned andactually experienced the reported outcome(s) as aresult of that intervention.The majority of the items relating to the selectionof the study sample (domain #2) were related toexternal validity. Although prospective orretrospective sample selection is more an issue ofinternal validity, the provision of explicit inclusionand/or exclusion criteria and the

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 6 List of quality domains and items#1 Background/context 1.1. Provision of background information1.2. Problem/question clearly stated b1.3. Study originality1.4. Relevance to clinical practice1.5. Rationale/theoretical framework#2 Sample definition and selection 2.1. Retrospective/prospective b2.2. Inclusion/exclusion criteria b2.3. Sample size b2.4. Selected to be representative b2.5. Baseline characteristics described#3 Interventions 3.1. Clear specification b3.2. Concurrent/concomitant treatment3.3. Feasibility of intervention#4 Outcomes 4.1. Clear specification b4.2. Objective and/or reliable4.3. Selection of outcomes for relevance, importance, side-effects#5 Creation of treatment groups a 5.1. Generation of random sequence b5.2. Concealment of allocation b5.3. How allocation occurred b,c5.4. Any attempt to balance groups by design b5.5. Description of study design5.6. Suitability of design5.7. Contamination#6 Blinding 6.1. Blind (or double-blind) administration b6.2. Blind outcome assessment b6.3. Maximum potential blinding used6.4. Testing of blinding#7 Soundness of information 7.1. Source of information about the intervention b7.2. Source of information about the outcomes b#8 Follow-up 8.1. Equality of length of FU d for the two groups? b8.2. Length of FU adequate?8.3. Completeness of FU b#9 Analysis: comparability 9.1. Assessment of baseline comparability b9.2. Identification of prognostic factors b9.3. Case-mix adjustment b#10 Analysis: outcome 10.1. Intention-to-treat analysis b10.2. Appropriate methods of analysis b10.3. Pre-specified hypotheses#11 Interpretation 11.1. Appropriately based on results b11.2. Assessment of strength of evidence b11.3. Application/implications b11.4. Clinical importance and statistical significance11.5. Interpretation in context#12 Presentation and reporting 12.1. Completeness, clarity and structure b12.2. Statistical presentation and reportinga Underlined domains denote ‘core’ domains.b Denotes items specified a prioric Items in italics are ‘core’ items (see text).d FU, follow-up.representativeness of the sample are externalvalidity issues. Issues relating to sample size mayassess the possibility that the study wasunderpowered to detect a clinically importanteffect or that the sample size was not determineda priori but by the point at which the study resultsbecame significant (an issue of internal validity).As the sample size is accounted for statistically27© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Evaluation of checklists and scales for assessing quality of non-randomised studieswhen performing a meta-analysis (and the studiesweighted accordingly), there is some debatearound whether it should additionally beconsidered as a criterion of quality. However,where studies are not formally pooled, as innarrative reviews, there is a case for including anassessment of the adequacy of sample size as partof the quality assessment process.Domains largely unrelated to study validityincluded reporting of the background and contextof the study (domain #1), the specification of theinterventions and outcomes assessed (domains #3and #4) and issues related to interpretation andpresentation (domains #11 and #12). Thesedomains were judged to be more related to qualityof reporting of a study.Authors’ quality items were allocated to our prespecifieditems as far as possible, but in some casesadditional items were added to accommodate allauthors’ items. At the analysis stage, the prespecifieditems were considered to be ‘key’ criteria.Four of these criteria were designated ‘core’criteria; these were the items that we consideredthe most relevant for the assessment of nonrandomisedstudies. A total of 18 items wereadded post hoc. Additional items relating to validityas opposed to reporting were as follows:concurrent/concomitant treatment; objectiveand/or reliable outcome assessment;contamination; testing of blinding; and adequacyof length of follow-up. Other items relating tointernal validity on a broader level included‘suitability of study design’ and ‘maximumblinding used’. The remainder of the additionalitems related more to presentation and reportingissues.Tool selection processOwing to the descriptive nature of the informationextracted, the data from each study were tabulatedand synthesised in a qualitative manner. It wasanticipated that a large number of qualityassessment tools and systematic reviews usingquality assessment would be identified. In order toreduce the number of tools that were discussed indetail, a primary selection criterion was adopted,as follows.‘Top’ toolsFor analysis purposes, a ‘good’ quality assessmenttool was deemed to be one that included prespecifieditems from at least five of the sixinternal validity domains. The pre-specified itemswere used both because they had been specified apriori and because they were judged to be moredirectly related to internal validity than themajority of items added subsequently. The degreeto which a tool met each of the six domains wasdisplayed using the star plot facility in thestatistical software package STATA, release 7.0(STATA Corp., College Station, TX, USA). A starplot is constructed by assigning each of the sixinternal validity domains to a separate axis(Figure 1). Because the domains did not all havethe same number of pre-specified items, each axiswas scaled between 0 and 1. A tool that coveredeach domain, and each item within that domain,would achieve a symmetrical ‘star’ shape.Creation of treatment groupsAnalysis:outcomeBlindingAnalysis:comparabilitySoundness ofinformationFollow-up28FIGURE 1 Sample star plot

Health Technology Assessment 2003; Vol. 7: No. 27The selected tools were described and broadlycompared with those tools which did not meet theprimary selection criterion, according to thenumber of times a tool had been used in oursample of reviews, the development of the tooland the purpose for which it had been designed.It was hypothesised that tools developedspecifically for use in a systematic review, andparticularly ones that were intended for theassessment of non-randomised studies, might bemore practical, more focused on issues relevant tothe assessment of non-randomised studies andmore likely to provide a means of comparingquality across studies.‘Best’ toolsThose tools which covered at least three of thefour core items were considered to be the ‘best’tools and were evaluated in more detail. Membersof the project team used each tool at least twice,on one of three non-randomised studies. The firstof these was a non-randomised trial of singleversus multiple mammary artery bypass forpatients undergoing coronary revascularisation, 61where two surgeons each primarily conducted oneof two types of surgery such that patient allocationoccurred according to the surgeon from which thepatient received their clinical consultation. Thesecond study was a retrospective concurrent cohortstudy of pneumococcal vaccination, whereallocation occurred according to clinician and/orpatient preference. 62 The final study was anexperimental before-and-after study on a singlegroup of patients with diabetic nephropathy toexamine the effect of restricting dietary protein onrenal failure. 63For each tool, the reviewer completed an appraisalform to indicate how long it took to complete thetool, the ease of use of the tool, whether any itemswere ambiguous or difficult to interpret or weremissing from the tool and any commentsregarding the tool’s suitability for use in asystematic review. The results of the appraisal arediscussed narratively and examples of the types ofquestions included by the tool’s authors areprovided.ResultsGeneral descriptionA total of 213 potential quality assessment toolswere identified. On closer inspection, 13 of thecited papers did not present tools for qualityassessment, and seven related to ‘levels ofevidence’ to categorise studies by design asopposed to investigations of quality. Full details ofthe remaining 193 tools are provided inAppendix 3, and selected features are presented inTable 7. Throughout, tools are referred to either bythe tool name or by the principal author. Eleventools developed by the authors of systematicreviews did not provide a list of the quality itemsassessed and could not be included in the contentanalysis. Sixty of the 182 tools that did report thequality items were selected as ‘top’ tools, coveringat least five of the six internal validity domains(Table 8). Of these, 14 met the criteria for ‘besttools’. Figure 2 provides a flowchart of the toolselection process.Only three of the identified tools wereunpublished, 64–66 with the remainder published injournals or book chapters. All three unpublishedtools were identified through contact with expertsin the field and were all selected as ‘best tools’.Almost three-quarters of the tools (73%) were‘new’ (or did not state the origin of the tool) andthe remainder were modifications of a singleexisting tool. The most commonly modified toolswere Chalmers and colleagues 51 and theMaastricht Criteria list, 67 the latter having beenpublished several times by different authors, 68–73but originating in 1989. 74,75Most (182) of the identified tools aimed to assessmore than one study design. Only 23 of theseincluded items specific to different designs; therest included generic items to be applied to alldesigns. Eight of the tools were designedspecifically to assess only RCTs but had been usedto assess non-randomised studies, including theChalmers scale 51 plus two modifications, 76,77 twoversions of the Maastricht Criteria List, 70,71 plusthree others. 46,78–80 The remaining four tools weredesigned specifically for cohort studies only[Anders, 81 Baker, 82 Critical Appraisal SkillsProgramme (CASP) 64 and Newcastle–Ottawa 66 ].Overall, around half of the tools were scales andhalf were checklists, although a higher proportionof the final sample of ‘best tools’ were checklistsrather than scales (79%). Of those scales thatreported the weighting scheme used, around onethirdimplemented an unequal weighting system.Most of those using unequal weighting did notdevelop an entirely new weighting scheme: of the‘top tools’, nine were derivatives of the Chalmersscale 51 and three were based on the MaastrichtCriteria list. 67 In spite of this, a variety of itemsreceived the highest weighting, includingrandomisation or the method of allocation used,allocation concealment, sample size and29© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Evaluation of checklists and scales for assessing quality of non-randomised studiesTABLE 7 Selected features of identified quality assessment toolsTool feature Best tools Top tools Other tools Total(n = 14) (n = 46) (n = 133) (n = 193)Publication statusPublished 11 (79%) 46 (100%) 133 (100%) 190 (98%)Unpublished 3 (21%) 0 0 3 (2%)Type of toolChecklist 11 (79%) 18 (39%) 75 (56%) 104 (54%)Scale 3 (21%) 28 (61%) 58 (44%) 89 (46%)If scaleEqual weighting 1 (33%) 14 (50%) 30 (51%) 45 (50%)Unequal weighting 2 (67%) 14 (50%) 16 (29%) 32 (37%)Not described 0 0 12 (20%) 12 (13%)Tool originNew tool 12 (86%) 29 (63%) 101 (76%) 142 (74%)Modified tool 2 (14%) 17 (37%) 32 (24%) 51 (26%)Tool developmentLiterature/consensus 6 (43%) 26 (57%) 41 (31%) 73 (37%)Survey/expert panel 3 (21%) 3 (7%) 5 (4%) 11 (5%)Not described 8 (36%) 17 (36%) 87 (65%) 112 (58%)Validity/reliabilityValidity 1 (7%) 3 (7%) 5 (4%) 9 (5%)Reliability 2 (14%) 22 (48%) 39 (29%) 54 (28%)Tool purposeStudy planning 0 2 (4%) 1 (1%) 3 (2%)Statistical analysis 0 2 (4%) 13 (10%) 15 (8%)Critical appraisal 5 (36%) 12 (26%) 22 (16%) 39 (20%)Guidelines 0 1 (2%) 0 1 (1%)Peer review 0 0 1 (1%) 1 (1%)Reporting 1 (7%) 1 (2%) 3 (2%) 5 (3%)Systematic review 8 (57%) 28 (61%) 93 (70%) 129 (65%)Number of itemsMedian (range) 22 (8–103) 17 (7–49) 10 (3–162) 12 (3–162)30description of the interventions. The basis for theweighting system adopted was rarely described.The method used to develop the tool was rarelyreported in detail. Most tools were based on someform of consensus, gleaned either from themethodological literature, existing qualityassessment tools or expert opinion. A total of 11tools used either surveys or panels of experts toselect the quality items and piloted and/orcirculated the list of items to experts for commentand revision. 46,66,83–91 Twenty tools gave noindication of how the items listed had beengenerated.A very small minority of tools attempted toestablish the validity of the tool (only four of thetop 60 tools). Of these, two found face and contentvalidity to be reasonable. 84,85 All four stated thatthey had examined criterion validity. Two of thesecompared their tools with that of Detsky andcolleagues (a reduced version of the Chalmersscale for assessing the quality of RCTs): 92 one 84found a significant difference in mean scores( p < 0.001), but a similar rank order of scores(Kendall’s W = 0.74); the other 93 did not presentthe results of the comparison. Downs and Black 85found a high correlation (Spearman correlationcoefficient 0.90) between the total score from theirtool (for both randomised and non-randomisedstudies) and that obtained for the Standards ofReporting Trials Group tool 94 (used onrandomised studies only). Linde and colleagues 95found similar results from their tool and that ofJadad and colleagues, 46 but did not carry out astatistical comparison.Twenty-four of the top 60 tools assessed inter-raterreliability. Two of these stated only that agreementwas mediocre or good and nine provided onlypercentage agreement between raters (range70–94%). Where provided (13 studies), kappa orother correlation coefficients were generally >0.75(n = 12), indicating good agreement. A few

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.TABLE 8 Details of top 60 quality assessment tools (14 ‘best’ tools denoted by shaded areas)Author a Originality b Type of No. of NRS Used in Tool Tool Tool 5 IV Core 5 domains andtool c items d items? e reviews? f purpose g validity h reliability i domains j items k 3 core items lAntczak, 1986 76 m; Chalmers, 1981 51 s u 30 n 2 7 na na Y 0Audet, 1993 176 m; Poynard, 1988 403 s e 17 n 1 7 na IRR Y 1Bours, 1998 73 m; van der Windt, 1995 71 s u 18 y n 7 na IaRR Y 1Bracken, 1989 104 n c 36 n n 6 na na Y 4 YCameron, 2000 83 n c 36 n n 7 na na Y 0Campos-Outcalt, 1995 177 n s e 7 n n 7 na na Y 1Carter, 1994 178 m; Chalmers, 1981 51 s u 12 n n 7 na na Y 0CASP, 1999 64 n c 10 n n 3 na na Y 3 YChalmers, 1981 51 n s u 36 n 15 3 na na Y 0Cho, 1994 84 m; Spitzer, 1990 105 s e 31 n n 3 FC, Cr IRR Y 2Cochrane MIG, 2002 179 m; Chalmers, 1981 51 s e 12 n CC 7 na na Y 1Coleridge Smith, 1999 97 n c 49 y n 7 na IRR Y 2Cowley, 1995 109 n c 17 n n 7 na na Y 3 YCuddy, 1983 180 n c 17 n n 3 na na Y 2Dawson-Saunders, 1990 98 n c 49 y n 3 na na Y 2de Oliveira, 1995 96 m; Chalmers, 1981 51 s u 30 n 1 7 na IRR Y 1de Vet, 1997 72 m; ter Riet, 1990 68 s e 15 n n 7 na IRR 67 Y 2Downs, 1998 85 n s u 27 n 3 3 FC, Cr IRR; T-R Y 3 YDuRant, 1994 99 n c 103 y n 3 na na Y 3 YFowkes, 1991 107 n c 23 n 1 3 na na Y 3 YFriedenreich, 1993 100 n c 29 y n 7 na na Y 2Gardner, 1986 181 n c 26 n n 2 na na Y 2Glantz, 1997 182 m; Chalmers, 1981 51 s u 20 n n 7 na na Y 0Gordis, 1990 101 n c 17 y 1 3 na na Y 2Greenhalgh, 1997 183 n c 11 n n 3 na na Y 1Gurman, 1978 184 n s u 14 n n 2 na na Y 1Guyatt, 1993 79 n c 12 n 1 3 na IRR 185 Y 0Hadorn, 1996 102 m; US Task Force, 1989 378 c 41 y n 4 na na Y 3 YKay, 1996 186 n s e 20 n n 7 na na Y 0Koes, 1991 70 m; ter Riet, 1990 68 s u 17 n 3 7 na IaRR Y 0Kreulen, 1998 187 m; Antzcak, 1986 76 s u 17 n n 7 na IRR Y 1Kwakkel, 1997 188 n s e 16 n n 7 na IRR Y 1Lee, 1997 189 m; Cook, 1979 50 s e 7 n n 7 na IaRR Y 2Levine, 1980 190 n s e 30 n 1 3 na na Y 1Linde, 1999 95 n s e 7 n 1 2/7 Cr na Y 0MacMillan, 1994 191 m; Chalmers, 1981 51 s u 9 n n 7 na IaRR Y 1Maziak, 1998 192 n s e 10 n n 7 na IRR Y 0Meijman, 1995 193 m; Fowkes, 1991 107 s e 11 n n 2 na no stats Y 1continuedHealth Technology Assessment 2003; Vol. 7: No. 2731

32TABLE 8 Details of top 60 quality assessment tools (14 ‘best’ tools denoted by shaded areas) (cont’d)Author a Originality b Type of No. of NRS Used in Tool Tool Tool 5 IV Core 5 domains andtool c items d items? e reviews? f purpose g validity h reliability i domains j items k 3 core items lMelchart, 1994 93 n s e 15 n n 7 Cr IaRR Y 0Miller, 1995 87 n s u 12 y 1 7 na IaRR Y 2Moncrieff, 1998 195 n s e 30 n n 3 na IRR; IC Y 1Morley 1996 196 m; Chalmers, 1981 51 s u 30 n 1 2/6 na IRR Y 2Mulrow, 1986 197 n c 13 n n 7 na IaRR Y 2Newcastle–Ottawa 66 n s u 8 y n 7 na na Y 3 YNicolucci, 1989 77 m; Chalmers, 1981 51 s u 16 n 1 7 na IaRR Y 0Reisch, 1989 111 n s e 57 n CC 1/3 na IRR 114 Y 3 YSalisbury, 1997 198 n c 45 n n 7 na na Y 2Schechter, 1991 199 n c 30 n n 3 na na Y 2Sheldon, 1993 200 n c 36 n n 3 na na Y 2Spitzer, 1990 105 n c 20 n 1 7 na na Y 4 YTalley, 1993 106 n c 29 n n 7 na na Y 0Thomas 65 n c 21 n n 7 na na Y 3 Yvan der Windt, 1995 71 m; Koes, 1991 70 s u 17 n 1 7 na IaRR Y 1Vickers, 1996 201 n c 12 n n 7 na no stats Y 1Vickers, 1995 110 n c 21 n n 3 na na Y 3 YWeintraub, 1982 108 m; Lionel, 1970 315 c 47 n n 1/2/3 na na Y 3 YWilson, 1992 103 n c 10 y n 7 na na Y 1Wingood, 1996 202 n c 16 n n 7 na na Y 0Wright, 1995 203 n c 13? n n 7 na na Y 0Zaza, 2000 86 n c 22 n n 7 na na Y 3 Ya Name of tool or principal author.b n, New tool; m, modification of existing tool.c c, Checklist; s, scale, u, unequal weighting scheme, e, equal weighting scheme.d Total number of items.e Did the tool include items specific to non-randomised studies? n, no; y, yes.fHas the tool been used in identified sample of systematic reviews?: n, no; CC, used in Cochrane reviews; or give number of reviews.g What purpose was the tool designed for? 1, list for planning a study; 2, list for assessing a statistical/methodological analysis; 3, list for evaluating a study when considering thepractical application of the intervention; 4, list for evaluating/grading study recommendations when producing recommendations/guidelines; 5, list for peer reviewing; 6, list forreporting a study; 7, list for assessing studies included in a systematic review.h Was an attempt made to establish tool validity? na, not assessed; FC, face and content; Cr, criterion; Co, content.iWas an attempt made to establish tool reliability? na, not assessed; IRR, inter-rater reliability; IaRR, intra-rater reliability; IC, internal consistency; T-R, test-retest; a, % agreementassessed.jWere at least 5 internal validity domains covered? Y, yes.k Number of core items covered.lWere at least 5 internal validity domains and at least 3 of the 4 key items covered? Y, yes.Evaluation of checklists and scales for assessing quality of non-randomised studies

Health Technology Assessment 2003; Vol. 7: No. 27Potential toolsn = 213Included toolsn = 193Excluded:levels of evidence: n = 7not quality assessment tools: n = 13‘top tools’ an = 60Excluded from contentanalysis: n = 11‘Best tools’ bn = 14Suitable for use in asystematic reviewn = 6a ≥5 internal validity domains metb ≥3 core items includedFIGURE 2 Flowchart of tool selection processstudies found the level of agreement to bedependent on the experience of the raters.de Oliveira and colleagues 96 achieved higheragreement for experienced raters, whereasColeridge Smith 97 found higher agreement fortwo methodologists combined rather thanclinicians alone or for a clinical/methodologistpairing. Other aspects of reliability were rarelyexamined.The majority of the identified tools were designedfor use in a systematic review (and most were infact published within the context of a systematicreview). Of the remainder, most were criticalappraisal tools, designed to assist the reader whenconsidering the usefulness of a particularintervention in their situation, rather than to beused to compare quality across studies.The number of items in the tools ranged from 3 to162, with a median of 12 items. Those toolsselected as ‘top tools’ or ‘best tools’ had a highermedian number of items compared with theunselected tools. Those with the largest number ofitems (>50) were more likely to have separatesections and questions according to study design(e.g. including items specific to RCTs, cohortstudies, cross-sectional studies).Possibly the most relevant aspect of these tools forapplication in practice is how well they coverquestions of internal validity and, in particular,whether they covered the four core items. Elevenof the 193 tools did not specify the items includedin the tools. Of the remainder, 60 tools covered atleast five of the six internal validity domains.Individual star plots demonstrating how well, or towhat extent, these tools covered each of the sixinternal validity domains are provided in Figure 3.These plots indicate a significant variation in howwell these domains were covered. Figures 4 and 5provide information on how often each domainwas missed and how well the tools covered eachdomain. The domain most commonly missed wassoundness of information (38%), with only 37% oftools including an item relating to whether theinterventions had been received by the studyparticipants and only 25% including an item onwhether or not the outcomes were attributable.All of the top 60 tools included one of the two33© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Evaluation of checklists and scales for assessing quality of non-randomised studiesAntzcakAudet Bours Bracken CASP Cameron Campos-Outcalt CarterChalmers Cho MSK CRG Coleridge Smith Cowley Cuddy Dawson-Saunders DownsDuRant Fowkes Friedenreich Gardner Glantz Gordis Greenhalgh GurmanGuyatt Hadorn Kay Koes Kreulen Kwakkel Lee LevineLinde MacMillan Maziak Meijman Melchart Miller Moncrieff MorleyMulrow Newcastle-Ottawa Nicolucci Reisch Salisbury Schechter Sheldon SpitzerTalley Thomas Vickers Vickers Weintraub Wilson Wingood WrightZaza de Oliveira de Vet Van der WindtCreation of groupsAscertainmentComparabilityBlindingFollow-upAnalysis of outcomeFIGURE 3 Top 60 star plots34pre-specified items in the follow-up domain (mostcommonly equality of length of follow-up betweengroups (88% of tools)).On average, about half of the items that we prespecifiedwere included in the tools (Figure 5).The domain with the best coverage wasblinding. The two core domains, creation ofgroups and comparability of groups, had onaverage 42 and 55% of possible pre-specifieditems, respectively.Six of the eight tools designed only for RCTs wereincluded in the top 60. These were the Chalmersscale 51 and its derivatives, 76,77 the two versions ofthe Maastricht Criteria List 70,71 and the JAMAUsers’ Guide checklist. 78,79 The Jadad 46 andMcMaster 80 tools did not include items in five ofthe six domains. In contrast, nine of the 23 toolsaimed at more than one study design 73,87,97–103and with specific items according to design, plustwo of the four tools designed for cohortstudies, 64,66 were included in the top 60 tools.

Health Technology Assessment 2003; Vol. 7: No. 27100%80%60%Internal validity domainsOther domains40%20%0%#5 Creation of groups#6 Blinding#7 Ascertainment#8 Follow-up#9 Comparability#10 Analysis#1 Background#2 Sample#3 Interventions#4 Outcomes#11 Interpretation#12 PresentationFIGURE 4 Proportion of top 60 tools that missed each domain100%90%80%Internal validity domainsOther domain70%60%50%40%30%20%10%0%#5 Creation of groups#6 Blinding#7 Ascertainment#8 Follow-up#9 Comparability#10 Analysis#1 Background#2 Sample#3 Interventions#4 Outcomes#11 Interpretation#12 PresentationFIGURE 5 Average proportion of pre-specified items met per domain (mean/maximum number of items)35© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Evaluation of checklists and scales for assessing quality of non-randomised studies36In terms of the four pre-specified core items, 15 ofthe 60 tools included none of the core itemsdespite covering at least five domains, 16 coveredone core item and 15 covered two items. Theremaining 14 tools covered at least three coreitems and were considered to be the ‘best’ tools inour sample, two of which covered all fouritems. 104,105 It is interesting that of the six toolsthat included items in all six internal validitydomains, 72,77,83,85,97,106 only one 85 included threeof our four core items. None of the tools designedonly for RCTs included three of the four coreitems.‘Best’ toolsFourteen tools were identified which covered atleast five of the six internal validity domains andthree of the four core items. Tables 9 and 10itemise the pre-specified items covered by eachtool.Amongst the top 14 tools, the internal validitydomain with the poorest coverage was analysis(four tools with zero items – CASP, 64 Fowkes, 107Newcastle–Ottawa 66 and Weintraub 108 ), followedby blinding (missed by Bracken 104 and Zaza 86 ) andascertainment (missed by Cowley 109 andHadorn 102 ). The item most commonly missed wasequal follow-up between groups (included by onlytwo tools – Bracken 104 and Downs 85 ). Only threetools asked about use of intention-to-treat analysis(Cowley, 109 Thomas 65 and Vickers 110 ).The two core domains were reasonably wellcovered. For the creation of groups domain, all ofthe tools except those specifically designed onlyfor observational studies (Bracken, 104 CASP 64 andNewcastle–Ottawa 66 ) included an item onrandomisation, but only two tools specificallyconsidered the use of allocation concealment(Downs 85 and DuRant 99 ). Of the four core items,the most commonly missed item was that relatingto how allocation occurred. Only eight toolsincluded this item. Ideally, we were looking for anitem that asked about how participants got intotheir respective groups, for example, was it byclinician or patient preference or was it spatial ortemporal assignment. All of the tools except forDowns, 85 DuRant 99 and Hadorn 102 included thesecond pre-specified item – balancing of groups bydesign.For the comparability of groups domain the twopre-specified items – identification of prognosticfactors and use of case-mix adjustment – weremissed by only two tools 109,111 and one tool, 108respectively. All of the tools except the CASP tool 64and the Newcastle–Ottawa tool 66 asked if baselinecomparability had been assessed.Our pre-specified items in the remaining sixdomains (Table 10) were, on the whole, not wellcovered, except perhaps for that relating to theselection of the study sample. Every tool includedan item about the representativeness of thesample, and only four did not ask about the studyinclusion/exclusion criteria (CASP, 64 Cowley, 109Newcastle–Ottawa 66 and Thomas 65 ). One of theitems in this domain that is related to internalvalidity – retrospective or prospective selection ofthe sample – was included by only two tools,Cowley 109 and Reisch. 111The remaining five domains concerning thequality of study reporting were not well covered bythe tools. The most commonly included item wasone that considered clear specification of theinterventions (nine tools). On the other hand,clear specification of the outcomes was included inonly five tools.Qualitative assessment of the ‘best’toolsOf the best 14 tools, eight were judged to beunsuitable for use in a systematicreview. 64,99,102,104,105,107,108,110 A description ofwhich of the core criteria they covered and ourassessment of them is provided in Appendix 4.In summary, their unsuitability was largely relatedto the fact that they were not designed for use in asystematic review of effectiveness: one waspublished to guide the reporting of observationalstudies; 104 five were intended to help in the criticalappraisal of research articles; 64,99,107,108,110 and onewas developed for an epidemiological review. 105Overall, these tools generally prompted somethinking regarding quality issues, but were notformatted in such a way as to allow an overallassessment of study quality or the comparison ofquality across studies. Some 64,108 did conclude witha more general item requiring a judgement on theoverall quality of the study, but little guidance wasprovided as to how this judgement should bemade. The Hadorn tool 102 was intended for use insystematic reviews, but the assessors queried theinclusion of, or phrasing of, several of the items.For example, the emphasis on drug trials and useof placebos was felt to be overly specific.Six quality assessment tools were judged to bepotentially useful for systematicreviews, 65,66,85,86,109,111 although in several casessome modifications would be useful. All but one of

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.TABLE 9 Selected features of identified quality assessment toolsNewcastle-Bracken 104 CASP 64 Cowley 109 Downs 85 DuRant 99 Fowkes 107 Hadorn 102 Ottawa 66 Reisch 111 Spitzer 105 Thomas 65 Vickers 110 Weintraub 108 Zaza 86#5 Creation of groups5.1. Generation of random ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓5.1. sequence5.2. Concealment of allocation ✓ ✓5.3. How allocation occurred a ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓5.4. Balance groups by design a ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓#6 Blinding6.1. Blind (or double-blind) ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓6.1. administration6.2. Blind outcome assessment ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓#7 Ascertainment7.1. Receipt of the intervention ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓7.2. Attributable outcomes ✓ ✓ ✓ ✓ ✓ ✓#8 Follow-up8.1. Equal follow-up between ✓ ✓8.1. groups8.3. Completeness of follow-up ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓#9 Comparability9.1. Baseline comparability9.1. assessed ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓9.2. Prognostic factors identified a ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓9.3. Case-mix adjustment a ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓#10 Analysis10.1. Intention-to-treat analysis ✓ ✓ ✓10.2. Appropriate analysis10.2. methods ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓a Items specfic to assessment of non-randomised studies.Health Technology Assessment 2003; Vol. 7: No. 2737

38TABLE 10 Other domains: reporting and external validityNewcastle-Bracken 104 CASP 64 Cowley 109 Downs 85 DuRant 99 Fowkes 107 Hadorn 102 Ottawa 66 Reisch 111 Spitzer 105 Thomas 65 Vickers 110 Weintraub 108 Zaza 86#1 Background1.1. Background information ✓ ✓1.1. provided#2 Sample2.1. Retrospective/prospective ✓ ✓2.2. Inclusion/exclusion criteria ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓2.3. Sample size ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓2.4. Representative ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓#3 Interventions3.1. Clear specification of ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓3.1. interventions#4 Outcomes4.1. Clear specification of ✓ ✓ ✓ ✓ ✓4.1. outcomes#11 Interpretation11.1. Appropriately based on ✓ ✓ ✓ ✓11.1. results11.2. Assessed strength of ✓ ✓ ✓11.1. evidence11.3. Application/implications ✓ ✓ ✓ ✓#12 Presentation12.1. Completeness, clarity, ✓ ✓ ✓ ✓11.1. structureEvaluation of checklists and scales for assessing quality of non-randomised studies

Health Technology Assessment 2003; Vol. 7: No. 27the tools in this group were specifically developedfor use in a systematic review, 65,66,85,86,109 theirmain advantage being that the questions andresponses were phrased in such a way as to forcethe reviewers to be systematic in their assessmentof individual studies, and ensure that betweenstudyquality can be compared.The length and complexity of the tools varied, butmost provided guides for completion of the toolsand three of the tools were considered ‘easy to use’in our assessment. 65,85,109 Although a certaindegree of judgement is required for completion ofany tool (including phrases such as‘appropriateness of allocation’, ‘trulyrepresentative’ and ‘adequately described’), theguides help the reader to interpret the questionsand response options. In several of thesetools, 66,85,109,111 the responses to individualquestions could be used to contribute to an overalljudgement of study quality, using either number ofitems fulfilled 66,109,111 or summary scores. 85 Two ofthe selected tools 65,86 used a ‘mixed-criteria’approach, including specific factual questionsabout the study design and also requiring moregeneral judgements as to the degree of biaspresent in a study.A summary of each of the six selected toolsfollows, with details of how they covered the fourcore items provided in Appendix 5.CowleyCowley 109 developed a quality assessment tool foruse in a systematic review of total hip replacement.Separate items were provided for RCTs,comparative studies and uncontrolled case series.The tool provides a list of 13 questions for thequality assessment of comparative studies, splitinto two sections: ‘key criteria’ (seven items) and‘other criteria’ (six items). No details of themethod of item generation were provided. Studieswere rated A, B or C, according to the number of‘key’ and ‘other’ criteria that were met. Key criteriaincluded the method of assignment of patients togroups, matching of groups for key prognosticvariables, appropriate statistical analysis andspecification of interventions and outcomes.Several items were topic-specific. No items forsample size and method of sample selection wereprovided. A mixture of items relating to bothreporting and methodological quality wereincluded. The authors did not attempt to establishthe validity or reliability of the tool, and did notprovide a guide to completion. However, we foundthe criteria quick and easy to apply, taking 5–15minutes to complete the checklist. With somemodifications, the Cowley tool was judged to besuitable for use in a systematic review.DownsDowns and Black 85 developed a scale to assess themethodological quality of randomised and nonrandomisedstudies. The tool was used in twosystematic reviews in our sample, 26,112 and appearsto have been developed for such use. It provides alist of 27 questions to measure study quality, splitinto four sections: ‘reporting’ (10 items); ‘externalvalidity’ (three items); ‘internal validity – bias’(seven items); and ‘internal validity – confounding(selection bias)’ (seven items). Epidemiologicalprinciples and methodological literature were usedto generate an initial version of the tool, whichwas then piloted by experienced epidemiologistsand statisticians and a revised version produced.The tool was found to be fairly comprehensive;however, questions regarding the allocationmechanism used relate only to randomisedstudies, and no items relating to baselinecomparability were included. Nevertheless, it wasfound to be easy to use, with yes/no/can’t tellanswers and clear descriptions of how to scoreitems. The tool authors found the validity andreliability of the tool to be reasonably high, exceptfor the three items on external validity. Althoughrelatively quick to complete (10–20 minutes), thetool is fairly long and a large number of questionsrelate to reporting as opposed to validity. TheDowns tool was judged to be suitable for use in asystematic review, although some modification,such as the addition of the missing items (e.g.baseline comparability), may be warranted.Newcastle–OttawaThe Newcastle–Ottawa tool 66 was designed for usein an epidemiological systematic review, and canbe used as either a scale or a checklist. It wasdeveloped using a Delphi process to definevariables for data extraction, was then tested onsystematic reviews and further refined. Separatetools were developed for cohort and case–controlstudies, although only the tool for cohort studies isconsidered here. The tool contains eight items,categorised into three groups: selection,comparability and outcome. For each item a seriesof response options are provided. For example, forthe consideration of the representativeness of theexposed cohort, response options include trulyrepresentative, somewhat representative, selectedgroup or no description of exposed cohort. A starsystem is used to allow a visual semi-quantitativeassessment of study quality, such that the highestquality studies are awarded a maximum of one39© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Evaluation of checklists and scales for assessing quality of non-randomised studies40star for each item within the selection andoutcome categories and a maximum of two starsfor comparability.Items relating to the appropriateness of theanalysis would make the tool more comprehensivein coverage. Further modifications would benecessary to adapt the tool for use in reviews ofeffectiveness of interventions rather than reviewsof causation. For example, an item on method ofallocation of participants to groups would beuseful. (Such modifications to the tool have in factbeen made for use in a systematic review ofarterial revascularisation. 113 ) No information wasprovided on the reliability or validity of the tool.Our reviewers found it relatively easy to use,taking only 5–10 minutes to complete. With theaforementioned caveats, the Newcastle–Ottawatool was judged to be suitable for use in asystematic review.ReischReisch and colleagues 111 developed their tool tofacilitate the evaluation of the design andperformance of therapeutic studies and to teachcritical appraisal skills. It is currently used by theCochrane Inflammatory Bowel Disease Group.The tool was based on ‘accepted researchstandards’ and lists a total of 57 items groupedinto 12 categories including study design, samplesize, randomisation and comparison groups. Thetool aims to evaluate any study design and is verycomprehensive in its coverage of internal validity.It is organised such that a systematic approach toanswering each question is ensured. Responseoptions include yes, no, unclear or unknown, andnot applicable.The criteria considered most important weredesignated as primary criteria (a total of 34 items).The selection of some of the items, however, maybe questionable regarding their importance forinternal validity. For example, statement of thepurpose of the study is more a reporting issuethan one of validity. On the other hand, otheritems selected are very relevant, including both‘randomisation claimed and documented’ and ‘useof either prognostic stratification prior to studyentry or retrospective stratification during dataanalyses’. The tool is also useful for reviewsincluding non-randomised studies in that it askswhether a randomised design would have beenpossible. However, some of the criteria are rathertoo specific to pharmaceutical studies and wouldrequire modification for more general use. Theinter-rater reliability of the tool was found to behigh when assessed in a later study, 114 especiallywhen only the primary criteria were considered(Pearson r = 0.99). However, we found the tool tobe rather long, taking approximately 25 minutesto complete. Overall the Reisch tool was judgedsuitable for use in a systematic review, although ashorter version would be more useable.ThomasThe tool produced by Thomas 65 was developed toassess the methodological quality of studies. Itaims to cover any study design and includes 21items separated into eight components: selectionbias, study design, confounders, blinding, datacollection methods, withdrawals and dropouts,intervention integrity and analysis. The method ofgeneration of the items included was not reported.A list of response options for each item isprovided. Following completion of the tool,reviewers are asked to give an overall rating of thestudy (strong/moderate/weak) for each of the firstsix components in order to provide a globalquality rating.This tool is able to deal with both randomised andnon-randomised studies, including items on bothrandomisation and control for confounders,although methods of allocation other thanrandomisation were not considered. The validityand reliability of the tool were not reported. Wefound it easy to use, taking 10–15 minutes tocomplete, with a comprehensive guide forcompletion provided. The Thomas tool wasjudged to be suitable for use in a systematicreview.ZazaThe tool by Zaza and colleagues 86 was designedfor use as part of a data collection instrument forsystematic reviews to inform the US ‘Guide toCommunity Preventive Services: SystematicReviews and Evidence-based Methods’. Items weregenerated from a review of methodologicalliterature, and expert opinion was solicited onpilot versions. The tool aims to cover any studydesign, with 22 quality items, grouped into sixcategories: descriptions, sampling, measurement,data analysis, interpretation of results and other. Itwas developed as part of a much largerstandardised data abstraction form and detailedinstructions for completion and cross-referencingto other parts of the data extraction form areprovided.The items included are not specific to anyparticular study design but are phrased in such away that they can be applied to any study design.The authors provide explicit decision rules and

Health Technology Assessment 2003; Vol. 7: No. 27explanation of responses to aid completion. Theconcept behind the tool seems a good one, but itis difficult to complete in isolation from the rest ofthe data extraction form. Despite the instructionsfor completion, some of the items may be toogeneric and difficult to interpret. For example,allocation of participants to groups is consideredin the ‘interpretation of results’ section under theitem ‘considering the study design, wereappropriate methods for controlling confoundingvariables and limiting potential biases used’. Thisitem aimed to cover randomisation, restriction,matching, etc. The complexity of the tool meantthat it took up to 30 minutes to complete. It maybe that the ease of use would increase withpractice. The Zaza tool may be suitable for use insystematic reviews but does require a goodunderstanding of validity issues.DiscussionA total of 213 potential quality assessment toolswere identified for inclusion in this review. Overallthe tools were poorly developed, with almost noattention to standard principles of scaledevelopment. 57 Almost without exception, theitems included in the tools were based on so-called‘standard criteria’ gleaned from methodologicalliterature, clinical trial or epidemiology textbooksor a review of other quality assessment tools. Mosttools did not provide a means of assessing theinternal validity of non-randomised studies, andseveral were aimed specifically at only randomisedtrials. Only 60 (30%) included items related to fiveout of six internal validity domains. Of these, 14were sufficiently comprehensive in their contentcoverage to be considered in detail. In order to beselected, these tools had to include at least threeof four pre-specified core items:●●●●how allocation occurredany attempt to balance groups by designidentification of prognostic factorscase-mix adjustment.These were selected as core items because for nonrandomisedstudies it is important to know, first,how study participants got into the interventiongroups, that is, was it by clinician or patientpreference, or was it according to spatial ortemporal factors and, second, what specific factorsinfluenced the selection of patients into eachgroup. Given that study investigators rarely reportthe latter information, the identification ofprognostic factors that influence outcome (asopposed to allocation) was included as a core item.However, covering these items does not necessarilymake the tools useful. For example, asking for adescription of the method of allocation 109,111 doesnot force a judgement about whether that methodwas appropriate or unlikely to introduce bias.Other than their content coverage, the top 14tools did not stand out from the remaining toolsin terms of their development, which was oftenvaguely reported, or the investigation of validityor reliability.A relatively informal assessment of the usefulnessof the top 14 tools identified six that werepotentially useful for systematicreviews. 65,66,85,86,109,111 The main advantage ofthese tools over the remaining eight was thephrasing of the items. On the whole, these forcethe reviewer to be systematic in their studyassessments and attempt to ensure that qualityjudgements are made in the most objectivemanner possible. Four of the tools attempted tocover most study designs using the samequestions, which could be the most usefulapproach for a systematic review whichincorporates several different study designs. Thisapproach did appear reasonably successful,although it may often be more appropriate tothink of quality issues on top of a study design‘hierarchy’. For example, there seems to be littlepoint in being overly concerned with blindinglevels when comparing an RCT with a before-andafterstudy. Furthermore, in many cases a fullassessment of study quality may require additionalcontext-specific questions to cover aspects ofexternal validity that would not be included in ageneric quality assessment tool, for example, itemsrelating to the quality of delivery of anintervention or quality of outcome measurements.Many of the tools, including some in the top14, 85,111 contained several items unrelated tomethodological quality. Although it is important todistinguish the quality of a study from the qualityof its report, one of the identified problems withscales for assessing RCTs is the inclusion of itemsrelating to issues such as reporting quality. 59Those tools which followed the lines of Cooper’s‘mixed-criteria’ approach, 45 requiring objectivefacts regarding a study’s design followed by aquality judgement (e.g. Thomas 65 ), may prove tobe the most useful for systematic reviews. Suchtools make as explicit as possible the factsregarding a study’s methods that should underliea reviewer’s judgement. Some tools were foundthat ignore judgements entirely and others41© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Evaluation of checklists and scales for assessing quality of non-randomised studiesprovoke thinking around quality issues but do notprovide a means of being systematic (e.g. tools tohelp critical appraisal of studies). Both of theseapproaches seem equally unacceptable.One of the main advantages of our review is thatwe identified a large number of quality assessmenttools, developed for use in a wide range of fieldsand published in a variety of formats. Although itwould not be possible to identify all availabletools, we did attempt to be as systematic aspossible. Our search covered a wide range ofdatabases and included both published andunpublished material. As a result, we are likely tohave identified a large proportion of availablequality assessment tools. In addition, all tools wereassessed in a systematic manner, by twoindependent reviewers.Nevertheless, several criticisms could be putforward. First, we could be criticised for beingover-inclusive as we did not require the tools tostate that their purpose was the methodologicalquality assessment of non-randomised studies. Werequired only that the tool had been or couldpotentially be used to assess such studies. Thisresulted in the inclusion of a wide range of tools,including some originally developed for RCTs. Wefeel that this approach can be justified as our aimwas to provide a comprehensive picture of hownon-randomised studies are being assessed andwhat other authors consider to be importantquality features. Furthermore, we had no a priorireason to assume that tools developed for RCTswould not be suitable for use with non-randomisedstudies, with or without adaptation.Second, the list of quality items against which weassessed the included tools was of our owngeneration, and as such, could be questioned. Ourstrategy was to take everything that is known aboutquality for randomised studies and add what weknow to be different about non-randomisedstudies. It should be noted that the aim of thetaxonomy was to provide an aid to data extractionof the identified tools as opposed to providing anexhaustive or comprehensive list of quality items.The alternative would have been to extract detailsof all the items from each quality assessment tooland then try to group them, but the sheer volumeof tools, and the use of varying terminology andquestion phrasing, would have made thisimpossible.Furthermore, the selection process that we used toidentify the ‘best’ tools (i.e. items in five out of sixinternal validity domains plus at least three of fourpre-specified core items) meant that those in thefinal sample of 14 would not necessarily have thebest coverage of the six domains. However, theseselection criteria were chosen first to reflect oura priori position that degree of selection bias is afundamental issue that needs to be addressed byquality assessment tools (hence the selection of thefour core items) and second because a morerestrictive policy, for example requiring items inall six domains, would have led to the finalassessment of only six tools for their ‘usability’ insystematic reviews.Another criticism that could be made relates toour assessment of the ‘usability’ of the best tools.This was based on their application to only threenon-randomised studies and could benefit fromrepetition on a larger sample of studies.Future research should include:●●●further appraisal of the quality criteria that weselectedconsideration of the creation of a new tool orthe revision of an existing onemore detailed examination of tool ‘usability’,for example, tools may be more or less usefulaccording to field or type of intervention.42

Health Technology Assessment 2003; Vol. 7: No. 27Chapter 5Use of quality assessment in systematic reviewsof non-randomised studiesIntroductionAs we have discussed in some detail in previouschapters, the principal difference betweenrandomised and non-randomised studies lies inthe latter’s considerable susceptibility to selectionbias. Concealed randomisation specificallyremoves the possibility of selection bias orconfounding in RCTs, i.e. any differences betweenthe groups are attributable to chance or to theintervention, all else being equal. 20 For nonrandomisedstudies, confounding between groupsis likely. Whatever the method of allocation, thereasons for the choice of an intervention can beinfluenced by subtle clues that are not easilyidentifiable but which may relate, for example, tothe patient’s prognosis. 3 Treatment selection maybe confounded by differences in case-mix, but alsoby use of concomitant healthcare interventions orvariation in the outcome assessment process, forexample where allocation is influenced bygeographical location or temporal differences. Anyassessment of the degree of selection biasintroduced by non-random methods of allocationis of primary importance for systematic reviews ofnon-randomised studies.Empirical evidence for the impact of assessingstudy quality on the results of systematic reviews islimited. However, work on RCTs generallyindicates that low-quality trials can lead to adistortion of true treatment effects. 20,21,59,115,116Furthermore, it has been shown that the choice ofquality scale can dramatically influence theinterpretation of meta-analyses, and can evenreverse conclusions regarding the effectiveness ofan intervention. 59 This lends support to the needfor careful choice of a quality assessment tool andfor consideration of the impact of study quality ona review’s conclusions.Recent guidelines for the reporting of meta-analysesof observational studies 117 recommend the“assessment of confounding, study quality andheterogeneity” to be clearly reported in themethods section of these reviews but moreoverthat “thorough specification of quality assessmentcan contribute to understanding some of thevariations in the observational studies themselves”.The results of quality assessment can be used in asystematic review in several ways, 92 includingforming inclusion criteria for the review orprimary analysis; informing a sensitivity analysisor meta-regression; weighting studies; orhighlighting areas of methodological qualitypoorly addressed by the included studies and theimpact of this on the review’s conclusions.The objectives of the study described in thischapter are to estimate the extent to which qualityassessment has been used in systematic reviews ofnon-randomised studies and can adequatelyidentify the sources of potential bias, and todescribe the methods used to incorporate theresults of the quality assessment into theconclusions of a review.MethodsStudy selectionWe sought systematic reviews evaluating theintended effect of an intervention and thatincluded at least one non-randomised study.Data sourcesThe NHS Centre for Reviews and DisseminationDatabase of Abstracts of Reviews of Effectiveness(DARE) was used to identify reviews. This is adatabase of quality-assessed systematic reviewsidentified by handsearching key major medicaljournals, regular searching of electronicbibliographic databases and scanning ‘grey’literature since 1994 (further details aboutDARE can be found athttp://agatha.york.ac.uk/darehp.htm). All reviewsentered in DARE up to December 1999 werescreened for inclusion. Further searches ofprimary databases with the aim of identifyingadditional systematic reviews not indexed inDARE were not carried out; however, any suchreviews identified from the other searches for theproject (see Chapters 3 and 4) were assessed forinclusion.43© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Use of quality assessment in systematic reviews of non-randomised studiesSystematic reviewsscreened1162Excluded:Diagnostic studies: 94RCTs only: 510Uncontrolled only: 47Reviews with NRS511No QA repoted343Quality assessed168Used existingQA tool69Modified existingQA tool28Developed ownQA tool71Tools usedNo. of reviewsChalmers: 6Maastricht Criteria List: 3Jadad scale: 3Gyorkos tool: 3Oakley: 3Haynes: 3Other (all

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 11 Breakdown of reviews including non-randomised studiesStudy designs included No. of reviews No. using quality Proportion using qualityassessment assessment (%)NRS only 32 5 16NRS + RCTs 216 99 46NRS + RCTs + uncontrolled 116 34 29NRS + uncontrolled 41 9 22Not clearly reported 106 21 20Total 511 168 33NRS, non-randomised study.least one RCT and in 87 reviews a term such as‘controlled trial’ was used to describe the studiesso that RCTs were not clearly distinguished fromnon-randomised studies (Table 11). Those reviewswhich included both RCTs and non-randomisedstudies were more likely to have conducted qualityassessment (conducted in 46% of cases) than thosewith non-randomised studies only (16%) or thosealso including uncontrolled studies. Full details ofthe review methods are provided in Appendix 6and review results in Appendix 7.Source of assessment toolsOf the 168 reviews claiming to have carried outquality assessment, 69 (41%) stated they had usedexisting quality assessment tools, 28 (17%) mademodifications to existing tools and 71 (42%)developed their own tool (Figure 6).Closer inspection of the papers cited in the 69reviews that used existing tools revealed that 10 ofthe tools (cited in 13 of the reviews) were not infact quality assessment tools, but weremethodological papers, textbooks or manuals onhow to conduct reviews 118–123 or design primarystudies. 50,124,125 Nineteen reviews used a variety ofversions of the ‘levels of evidence’ frameworkoriginally developed by Sackett 126 and did notassess quality beyond identifying the study design.Amongst the remaining 37 reviews, the mostfrequently used tools were originally developed forthe assessment of RCTs (Figure 6): the Chalmersscale, 51 or modifications of it 76,77,92,96 (six reviews);the Maastricht Criteria List 69,70 (three reviews);and the Jadad scale 46 (three reviews). Othercommonly used tools were ones developed for usein public health 127 (used in three reviews) and atool developed by Oakley and colleagues 128developed for and used by the same authors forreviews of the health promotion literature (threereviews).Of the 28 reviews that modified existing tools,only one 26 reported details about how the tool(originally published by Downs and Black 85 ) hadbeen modified and also attempted to examine thevalidity of the tool. Twelve of these reviewsreported inter-rater reliability, all producing kappacoefficients of >0.70 or percentage agreement ofat least 80%.In the review by MacLehose and colleagues, 26 theitems considered to be the most important andunambiguous ‘quality items’ were identified bythree authors and were categorised into one offour dimensions: quality of reporting, internalvalidity, susceptibility to confounding and externalvalidity. The internal consistency of the individualitems contributing to the first three of thesedimensions was described using Cronbach’s α.The resulting low α values indicated thatindividual items were poorly correlated with thesum of scores for the items, suggesting that theitems were not assessing a homogeneous aspect ofquality. An attempt was also made to validate thefour quality dimensions and the sum of the fourdimensions by investigating predictors of qualityscores. Study design, intervention and theinteraction of study design and intervention wereentered into the regression model. For totalquality scores, both cohort and case–controlstudies had significantly poorer quality than RCTs.There was no independent effect of interventionand no interaction of intervention and studydesign. Case–control studies scored more highlythan cohort studies.For the 71 tools developed by review authors, nodetails of the tool development were provided andnone justified the choice of quality items. Twentyonereviews examined inter-rater reliability andfound similar levels of agreement to thosediscussed previously. In addition, two furtherreviews 93,129 addressed the validity of their tools.45© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Use of quality assessment in systematic reviews of non-randomised studiesTABLE 12 Quality assessment in systematic reviews: coverage of internal validityProportion of Proportion of Proportion ofNo. with 1 reviews reviews claiming reviews includingpre-specified specifying items QA (%) NRS (%)items (%) (n = 124) (n = 168) (n= 511)Coverage of individual domainsCreation of groups 103 83 61 20Blinding 78 63 46 15Soundness of information 35 28 21 7Follow-up 89 72 53 17Comparability 68 55 41 13Outcome 46 37 27 9Summary of domains covered5 or 6 domains 34 27 20 7All 6 domains 11 9 7 2Summary of core items coveredNumber of core items met:1 62 50 37 122 19 15 11 43 5 4 3 14 0 0 0 0Number meeting each core item:5.3. How allocation occurred 24 19 14 55.4. Any attempt to balance groups by design 17 14 10 39.2. Identification of prognostic factors 12 10 7 29.3. Case-mix adjustment 33 27 20 7Reviews using ‘best’ tools:5 domains and 3 core items 3 2 2 1QA, quality assessment; NRS, non-randomised study.46One 93 looked at criterion validity but did notpresent the results of the comparison. Theother 129 used the judges panel technique toexamine face and content validity. The results ofthis were not reported in detail, only that thepreliminary list of over 200 constructs was reducedto 71 critical factors.Twelve reviews in the sample (eight of themodified tools and four developed by reviewauthors) did not report the quality items assessedby the authors.Content of assessment toolsThe quality criteria included in the assessmenttools used could be identified for 124 reviews andthese were examined in more detail (Table 12).The majority of these reviews (83%) included atleast one item relating to the ‘creation of groups’domain; 72% considered ‘follow-up’ and looked at63% ‘blinding’; and 55% included items relating tothe ‘comparability of groups’ domain. Less thanhalf of the 124 reviews considered ‘analysis ofoutcome’ or ‘soundness of information’. Only 34reviews contained items in at least five out of sixinternal validity domains and 11 contained itemsin all six domains.Fewer than 21% of all reviews that included nonrandomisedstudies addressed any one of the sixinternal validity domains (see final column of Table12), 5% assessed five internal validity domains and2% assessed six domains. Looking more closely atthe four quality items that particularly distinguishnon-randomised studies from RCTs, only 62reviews (12%) assessed at least one of the four coreitems, 19 of which (4%) assessed two of the itemsand only five (1%) looked at three of the four. Noreviews assessed all four items and 449 (88%)reviews addressed none of the four. Of the fouritems, reviews were most likely to consider whetherthe study had conducted any case-mix adjustment(33 reviews, 7% of the sample), how the allocationhad occurred (24 reviews, 4.7%), whether anymatching had been used to balance groups (17reviews, 3%) and whether all important prognosticfactors had been identified (12 reviews, 2%). Onlythree reviews 109,112,130 used one of what we judged

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 13 Reporting of quality assessment resultsNo. of studies per review Reviewsclaiming QA (%)No. of reviews Median Mean Range (n = 168)QA results per item 50 14 25 5–211 30Overall quality score per study 25 18 30 5–107 15No. of studies meeting each criterion 16 67 66 13–177 10Narrative discussion 21 21 34 4–224 13Average quality score across studies 12 30 41 9–106 7Levels of evidence 9 24 23 8–39 5Other 8 45 59 10–194 5Not reported 27 46 76 9–474 16QA, quality assessment.TABLE 14 Method of incorporating quality assessmentNarrative reviews Meta-analyses All reviews using QASynthesis method n (%) n (%) n (%)Narrative 76 (84) 11 (14) 87 (52)Quantitative 4 (4) 57 (74) 61 (36)Not reported 11 (12) 9 (12) 20 (12)Total 91 (100) 77 (100) 168 (100)QA, quality assessment.to be the ‘best’ tools in our sample, 85,109 that isincluding items in at least five out of six internalvalidity domains and three out of four core items.Reporting of study quality insystematic reviewsThe various methods used to report the results ofthe quality assessment are given in Table 13. Thesevaried from a detailed listing of the qualityassessment results by item (30%) to reporting onlythe average quality score across all of the includedstudies (7%). In 27 cases (16%), the qualityassessment results were not reported at all.Use of study quality in systematicreviewsThe means by which the results of the qualityassessment were considered in the study synthesesare reported in Table 14, split by whether thereview used only a narrative synthesis or includeda meta-analysis. Overall, over half of the reviews(87/168) discussed the results of the qualityassessment narratively. Generally, the authorsprovided a broad statement of the overall qualityof included studies (often inferring that thestudies were of too poor quality to allow firmconclusions to be drawn), although someattempted a narrative comparison of the results ofbetter studies compared with weaker studies. Afurther 12% of all reviews did not incorporate theresults of the quality assessment into the reviewsynthesis.Around three-quarters of the meta-analyses in thesample (57/77) attempted to incorporate theresults of the quality assessment in a quantitativeway, although the implications of study qualitywere not always discussed (Appendix 7). Half(28/57) investigated the impact of individualquality components, 13 looked for any associationbetween quality score and effect size and theremainder used a variety of techniques, includingquality weighting or quality thresholds.DiscussionWe found that 67% of systematic reviews thatincluded non-randomised studies either did notconduct, or failed to report, any form of qualityassessment. The remainder used a variety ofquality assessment tools, including several thatwere explicitly developed for the assessment ofRCTs. In a substantial proportion of cases (14%),review authors developed the tools themselves butgave very limited details regarding the method ofdevelopment.Our analysis of the content of the qualityassessment tools used revealed that the majority of47© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Use of quality assessment in systematic reviews of non-randomised studies48the 168 reviews that claimed to have undertakenquality assessment did not comprehensively assessthe internal validity of the included studies. Only34 reviews (20%) included at least one quality itemin five out of six internal validity domains andonly 62 (37%) assessed at least one of the four keyareas that distinguish non-randomised fromrandomised studies. Only three reviews (2%) usedquality assessment tools that we judged to paysufficient attention to key issues of selection biasfor non-randomised studies.Most of the reviews that assessed study qualityreported the results in some form, although in lessthan one-third of cases were the results reportedper study. This seems to be partially related to thenumber of studies per review – those with fewerstudies were more able to present detailed results.A minority of reviews that assessed quality (12%)did not go on to consider the quality assessmentresults in the study syntheses; however, mostprovided a narrative discussion of study qualityand its implications.Those reviews that attempted to incorporate studyquality into a quantitative synthesis did so in avariety of ways. Most included a wide variety ofstudy designs, but the numbers of primary studiesincluded were not large enough to allow thedegree of bias introduced by variations in qualityto be clearly identified; the impact of quality wasconfounded by differences in study design.Our review has revealed that the conduct ofsystematic reviews that include non-randomisedstudies with respect to quality assessment is aspoor as, if not worse than, that found by Moherand colleagues in 1999 for meta-analyses ofRCTs. 131 Moher and colleagues also found thattrial quality was not assessed in most metaanalyses(48% compared with 67% of reviews inour study) and that when it is assessed, theassessments are obtained with non-validated tools.The infrequency of adequate quality assessmentmay in part occur owing to the lack of empiricalevidence and controversy concerning the biasesthought to act in non-randomised studies, andconfusion both about what quality items to assessand what quality assessment tool to use. This isperhaps supported by our finding that thosereviews that included both RCTs and nonrandomisedstudies were much more likely to haveconducted quality assessment than those that didnot include RCTs or also included uncontrolledstudies.However, the absence of agreed criteria forassessing quality has not stopped reviews of nonrandomisedstudies of healthcare interventionsbeing carried out and their results being used toinform treatment and policy decisions. Given theclear evidence that inadequate randomisationprocedures severely bias the results of RCTs, itseems reasonable to predict that non-randommethods of allocation are equally if not moreopen to selection bias than concealed allocation.Where randomised evidence is unavailable,the potential for bias and resultinguncertainty inherent in estimates based onnon-randomised evidence should be stronglyemphasised and evaluated through qualityassessment.A particular strength of our review was theavailability of a large number of systematic reviewsfor assessment provided via the DARE database.This database is fed by monthly, and in some casesweekly, extensive literature searches of a widerange of databases [such as Current Contents,MEDLINE and Cumulative Index of Nursing andAllied Health Literature (CINAHL)] publishedsince 1994. Systematic reviews have to meet acertain standard of methodological quality beforebeing included on the database. This in turnmeans that the reviews in our sample are ofhigher quality than many that are published, sothat the situation in practice may be even worsethan we have demonstrated here. In the past,the process of assessing reviews for inclusion inDARE and the subsequent writing of structuredabstracts for each review has also meant thatthere was some time lag before reviews wereloaded on to the database. The majority ofreviews in our sample were published prior to1999 and it is possible that reviewers are nowmore likely to conduct and report qualityassessment of non-randomised studies than theyhave been in the past.Nevertheless, reviewers who include nonrandomisedstudies in their systematic reviewsshould be aware of the fundamental need toaddress the potential for selection bias in thesestudies (and also all of the other quality issues thataffect all study designs), and should consider theimpact of these biases in the synthesis of studiesand, in turn, in their conclusions. In turn, users ofsystematic reviews should be careful not to overinterpretthe results of reviews that included nonrandomisedstudies. If biases have not beenassessed then conclusions may be invalid orunjustified.

Health Technology Assessment 2003; Vol. 7: No. 27Chapter 6Empirical estimates of bias associated withnon-random allocationIntroductionIn Chapters 4 and 5, many aspects of study qualitythat may be related to bias in non-randomisedstudies were identified. Two quality items wereemphasised that are specific to non-randomisedstudies rather than RCTs: bias introduced by nonrandomallocation and the use of case-mixadjustment methods. In this chapter, we report anempirical investigation into the impact of nonrandomallocation. In Chapter 7 we report asimilar investigation into the benefits of case-mixadjustment.Eight reviews were described in Chapter 3 thatattempted to evaluate the importance ofrandomisation. Our appraisal, however, identifiedproblems with their methodology, especially withregard to control for confounders (metaconfounding)and their inability to identify biasesother than those acting in a systematic manner.For our evaluation, we have chosen a differentmethodology – resampling – to overcome theseissues. By selectively sampling treated and controlparticipants from a completed large randomisedtrial it is possible to make comparisons betweengroups of individuals who did and did not receivethe intervention, but where the treatmentallocation was based not on randomisation but ona non-random mechanism. Effectively we areanswering the question, ‘what results would thetrialists have obtained if they had not comparedtheir treated participants with random controls butwith controls selected using a non-randomprocess, all other aspects of study design beingkept the same?’.Two classical non-randomised designs can beconstructed from the data of a multicentre trial:first, a concurrently controlled cohort study inwhich treated patients from one centre arecompared with an untreated control group from adifferent centre recruited during the same timeperiod; and second, a historically controlled studywhere the untreated controls are selected from thesame centre but from a period before the treatedpatients were recruited. The results from suchstudies can be compared directly with the resultsof the RCT in a single centre. Repeating thecomparisons using replacement samplingmethodology with data from many centres cangenerate distributions of results for randomisedand non-randomised studies. These distributionscan be compared to describe the impact of bothsystematic and unpredictable components of biasassociated with the allocation mechanism. Thecomparisons between study designs are based onthe same patient data, and hence across repeatedsamples will be unconfounded.We report the results of such resampling exercisesfor two large multicentre trials: the InternationalStroke Trial (IST) 132 and the European CarotidSurgery Trial (ECST). 133 The IST comparedmedical treatments in stroke patients to preventlong-term death or disability. The ECST was atrial of a surgical treatment in patients at risk ofstroke to prevent stroke or death. Full descriptionsare given in Appendix 8.Note that because of the adaptations made to thetrials for the purposes of the empiricalinvestigation (such as grouping of centres,omission of data from centres and periods, andselection of particular time-points for follow-up),the results reported here differ from those of theoriginal publications. 132,133 Notably, the full reportof the ECST trial reports treatment benefit inparticipants with high-grade stenosis, whereas thesimplified data set that we consider presents afinding of an average harmful effect. The originalpublications should be referred to for theappropriate analyses of these trials. 132,133MethodsGeneration of non-randomised andrandomised studiesThe resampling was undertaken in a similar wayfor both the IST and ECST. First, participantswere grouped into regions as described inAppendix 8. Secondly, participants were classifiedaccording to whether they were early or laterecruits to the trial, depending on whether theywere recruited before or after a time-point midway49© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical estimates of bias associated with non-random allocation50through the recruitment of the trial. Figure 7illustrates this structure for the region-basedanalysis of the IST trial. Participants are classifiedaccording to the treatment they received (T or C),whether they were recruited in the first (subscriptB for before) or second (subscript A for after) halfof the trial, and according to the region number(subscripts 1–14).Historically controlled studies were constructedwithin each region by sampling participants whowere recruited to the control group during thefirst half of the trial and participants who wererecruited to the treatment group during thesecond half of the trial. For example, in NorthItaly, control participants were selected from cellC B1 and treated participants from cell T A1 .Concurrently controlled studies were constructedby selecting a region from which treatedparticipants were drawn and choosing controlparticipants from another region chosen atrandom. Participants were sampled regardless ofthe recruitment period. For example, aconcurrently controlled comparison could begenerated between treated patients in Scotland(sampling from cells T B4 and T A4 ) and controlpatients in Spain (sampling from C B12 and C A12 ). Itwas expected that no average bias would beobserved in large samples of concurrentlycontrolled studies generated in this manner asthere would be corresponding studies comparingtreated patients from Spain with control patientsfrom Scotland.The IST recruited sufficient participants forsamples of 100 treated and 100 controls to besampled from each region for each design. For theECST only samples of 40 treated and 40 controlswere drawn as sample sizes in regions were lower.In all instances random samples were drawn usingreplacement sampling techniques. 134 This partialbootstrap process ensures that the variabilitybetween samples is as close as possible to thatwhich would have been obtained if sampling froman infinite (or very large) population.For the purpose of comparing distributions ofresults, randomised trials of the same size as thenon-randomised studies were constructed usingthe same resampling methodology. RCTs forcomparison with historically controlled studieswere constructed by comparing treated patientssampled from the second half of the trial (thesame treated sample as selected for the HCT) withcontrol patients from the same region alsosampled from the second half of the trial. Forexample, a North Italy RCT for comparison with aNorth Italy HCT was constructed by samplingtreated participants from T A1 and controlparticipants from C A1 . RCTs for comparison withconcurrently controlled studies were constructedby comparing treated and control patientssampled from the same region regardless of therecruitment period. For example, a North ItalyRCT for comparison with concurrently controlledstudies was constructed by sampling treatedparticipants from T B1 and T A1 and controlparticipants from C B1 and C A1 (Figure 7).Historically controlled studies were constructed ineach of the 14 regions of the IST analysis and theeight regions of the ECST analysis. The wholeresampling process was repeated 1000 times inboth studies, generating results of 14,000 IST and8000 ECST historically controlled studies, and thesame number of comparable RCTs.Concurrently controlled studies wereconstructed for each of the 14 regions of the ISTanalysis and the eight regions of the ECSTanalysis. In addition, in an attempt to limit somesources of variability, concurrently controlledstudies were constructed using IST data from 10cities in the UK (see Appendix 8). All resamplinganalyses were repeated 1000 times, generating14,000 IST and 8000 ECST concurrentlycontrolled region-based comparisons and 10,000IST concurrently controlled UK city-basedcomparisons, all with the same number ofcomparable randomised trials.Statistical analysesThe focus of the statistical analysis was ondescribing and comparing the location (average)and spread (variability) of the treatment effectsobserved in studies of different designs. Treatmenteffects were calculated using standard methodscommonly reported in medical journals, andinterpreted according to their statisticalsignificance assessed at the 5% level usinga two-sided test.Observed outcomes of the resampled trialparticipants for each non-randomised andrandomised study were tabulated, and thetreatment effect expressed as an odds ratio (OR)with 95% confidence interval. For both IST andECST, OR < 1 indicate that the experimentaltreatment (IST, aspirin; ECST, carotidendarterectomy) had better outcomes thancontrols. The effect was deemed to be statisticallysignificant if the confidence interval excluded thenull hypothesis of no treatment effect (OR = 1).

Health Technology Assessment 2003; Vol. 7: No. 27Region Trial allocation Recruitment period a1st half2nd halfNorthern ItalyAspirinAvoid aspirinT B1C B1T A1C A1Central ItalyAspirinAvoid aspirinT B2C B2T A2C A2Southern ItalyAspirinAvoid aspirinT B3C B3T A3C A3ScotlandAspirinAvoid aspirinT B4C B4T A4C A4Northern England and WalesAspirinAvoid aspirinT B5C B5T A5C A5Southern EnglandAspirinAvoid aspirinT B6C B6T A6C A6AustraliaAspirinAvoid aspirinT B7C B7T A7C A7The NetherlandsAspirinAvoid aspirinT B8C B8T A8C A8New ZealandAspirinAvoid aspirinT B9C B9T A9C A9NorwayAspirinAvoid aspirinT B10C B10T A10C A10PolandAspirinAvoid aspirinT B11C B11T A11C A11SpainAspirinAvoid aspirinT B12C B12T A12C A12SwedenAspirinAvoid aspirinT B13C B13T A13C A13SwitzerlandAspirinAvoid aspirinT B14C B14T A14C A14a The IST recruitment period was split at 15 January 1995FIGURE 7 Resampling structure for IST used to generate non-randomised and randomised studies51© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical estimates of bias associated with non-random allocationRegionHistorical controlOdds ratio(95% CI)Northern ItalyCentral ItalySouthern ItalyScotlandNorthern England and WalesSouthern EnglandAustraliaThe NetherlandsNew ZealandNorwayPolandSpainSwedenSwitzerland1.58 (0.89 to 2.78)1.92 (1.09 to 3.38)0.84 (0.47 to 1.50)1.51 (0.86 to 2.66)1.00 (0.57 to 1.74)0.70 (0.40 to 1.22)0.45 (0.26 to 0.81)0.66 (0.37 to 1.16)1.00 (0.57 to 1.75)0.50 (0.29 to 0.88)1.00 (0.55 to 1.83)2.21 (1.18 to 4.15)0.79 (0.40 to 1.55)0.92 (0.42 to 2.03)0.3 1 2 3Average odds ratio: 0.97Standard deviation of log (odds ratios): 0.47Significant results: 2/14 show benefit, 2/14 show harmFIGURE 8 Illustrative results from one resampling iteration of historically controlled studies resampled from the IST. CI, confidenceinterval52Distributions of results for each design were firstplotted using dotplots. Location was described bycalculating the ‘average’ OR as the exponential ofthe mean of the log OR. The ratio of estimates ofaverage ORs between the RCTs and nonrandomisedstudies provided an estimate ofsystematic bias. Figure 8 shows the results of thefirst 14 historically controlled studies generatedfrom the IST data set, with an average log OR of–0.03 and standard deviation (SD) of 0.47. Theestimates of SD were averaged across the 1000sample repeats to summarise the spread of resultsfor each study design. As the sample size andaverage event rate are the same for randomisedand non-randomised studies, differences in spreadcan be wholly attributed to unpredictablity in thesize and direction of the bias introduced throughnon-random allocation. The difference in spreadwas expressed as the ratio of the SD of log OR fornon-randomised studies compared with that forRCTs. This ratio estimates how many times morevariable results from non-randomised studies arethan results of RCTs.To investigate the likely conclusions of the studies,the percentage reaching the conventional 5% levelof statistical significance was also tabulated.Results showing significant benefit were reportedseparately from results showing significant harm.ResultsConcurrently controlled studiescompared with randomised controlledtrialsThe overall results of the resampled concurrentcontrolled studies and RCTs are reported inTable 15. As expected, the concurrent controlledstudies from the IST showed no average systematicbias (the average OR, of concurrently controlledstudies and RCTs were very similar), but increasedvariation (the variability of the results ofconcurrently controlled studies were higher thanthose of RCTs). Owing to this increased variability,more concurrently controlled studies werestatistically significant than were RCTs. Concurrentcontrolled studies from the ECST showed only asmall increase in variability and no systematic bias.Results from the IST region-based analysisThe results of the 14,000 concurrently controlledstudies generated from the IST data set areplotted in Figure 9, together with results of 14,000

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 15 Comparisons of results of RCTs with results of concurrently controlled non-randomised studies based on analyses of 1000sets of resampled studiesPercentage of studies with statisticallysignificant results showing beneficial orharmful effects of the intervention (p < 0.05)Average OR SD of log OR Beneficial Harmful TotalIST – 14 regionsRCTs 0.91 0.34 7 2 9Concurrent controls 0.91 0.85 29 21 50Ratio of SDs2.5-foldIST – 10 UK citiesRCTs 1.01 0.49 7 6 13Concurrent controls 1.02 0.90 22 23 45Ratio of SDs1.8-foldECST – 8 regionsRCTs 1.08 0.68 3 5 9Concurrent controls 1.08 0.69 3 6 9Ratio of SDs1.01-fold5020105Odds ratio210.50.20.10.050.02RCTsCCsFIGURE 9 Comparison of distributions of results of 14,000 concurrently controlled studies (CCs) and 14,000 RCTs resampled from 14regions within the ISTRCTs generated from the same data. Althoughthere was no difference in average results, theincrease in variability of results in concurrentlycontrolled trials is clear. The SD of the distributionwas 2.5 times that of the RCTs (Table 15). Some50% of the concurrently controlled studies gavestatistically significant results compared with 9% ofthe RCTs, with 29% showing significant benefit ofaspirin and 21% showing significant harm.Results from the IST UK city-based analysisThe results of the 10,000 concurrently controlledstudies generated from the UK IST data areplotted in Figure 10, together with results from the53© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical estimates of bias associated with non-random allocation5020105Odds ratio210.50.20.10.050.02RCTsCCsFIGURE 10 Comparison of distribution of results of 10,000 concurrently controlled studies and 10,000 RCTs resampled from 10 UKcities within the ISTcomparable RCTs. Again, the increase in variabilityof results for concurrently controlled studies wasevident, the distribution of results being 1.8 timeswider than for RCTs, slightly less than for theregional IST comparisons (Table 15). Some45% of concurrently controlled results werestatistically significant compared with 13% ofRCTs, the significant results being evenlydistributed between benefit (22%) and harm(23%).1002020105Odds ratio210.50.20.10.050.020.01RCTsCCs54FIGURE 11 Comparison of distribution of results of 8000 concurrently controlled studies and 8000 RCTs resampled from eight regionswithin the ECST

Health Technology Assessment 2003; Vol. 7: No. 27Results from the ECST region-based analysisFigure 11 displays results of 8000 concurrentlycontrolled comparisons and 8000 RCTs generatedfrom the ECST data set. The pattern of results ofconcurrent controlled studies are similar to thoseof RCTs. The concurrently controlled comparisonswere not more variable than the RCTs; theirdistribution was only 1.01 times as wide as theRCTs (Table 15); 9% of both concurrentlycontrolled studies and RCTs were statisticallysignificant.Historically controlled studiescompared with randomised controlledtrialsTable 16 shows results of comparisons ofhistorically controlled studies and RCTs sampledfrom the IST and ECST. The data from thehistorically controlled studies generated from thetwo trials display different patterns.Results from the IST region-based analysisIn the overall results of the historically controlledstudies from the IST there was no evidence ofsystematic bias. However, there was an increase inunpredictability in study results; the distribution ofhistorically controlled results was 20% wider thanthat for the RCTs (Table 16). This increase isdiscernible in Figure 12, where the results of the14,000 historically controlled studies and 14,000comparable RCTs are displayed. The increasedunpredictability increased the percentage ofstudies deemed statistically significant from 11 to20%, with increases in the number of findings ofboth statistically significant benefit and harm.Table 17 presents the same results broken downaccording to region. The results disaggregated atthis level appeared rather different. For eachregion there was little evidence of an increase inunpredictability (the SDs of the historicallycontrolled studies were all within 7% of the SD ofthe RCTs). However, there was evidence ofsystematic bias within many of the regions,although the magnitude and direction of the biasvary. For example, in Scotland the average OR forhistorically controlled studies was 1.23 comparedwith an average OR of RCTs of 0.78. In contrast toScotland, in Sweden the average OR fromhistorically controlled studies was 0.44 comparedwith 0.94 from the RCTs, 81% of historicallycontrolled studies concluding statisticallysignificant benefit. When aggregated across theregions, these varying systematic biases on averagecancel out and manifest as an increase inunpredictability.Results from the ECST region-based analysisThe overall results of the historical controlledstudies sampled from the ECST analyses (Table 16)showed a large systematic bias, but with littleincrease in unpredictability. Whereas the averageOR in the 8000 RCTs was 1.23, the average OR ofthe 8000 historically controlled studies was 1.06.This difference is noticeable in the comparison ofthe distribution of the results of the studies inFigure 13. The results of the two study designsalso show similar levels of variability. Thesystematic bias impacted on statistical significanceby shifting the distribution of results in onedirection: 6% of the historically controlled studiesconcluding significant benefit from carotidsurgery compared with 4% of the RCTs, whereas10% of the historically controlled studiesconcluded significant harm compared with12% of the RCTs.TABLE 16 Comparisons of results of RCTs with results of historically controlled non-randomised studies based on analyses of 1000 setsof resampled studiesPercentage of studies with statisticallysignificant results showing beneficial orharmful effects of the intervention (p < 0.05)Average OR SD of log OR Beneficial Harmful TotalIST – 14 regionsRCTs 0.89 0.35 9 2 11Historical controls 0.88 0.44 16 4 20Ratio of SDs1.2-foldECST – 8 regions aRCTs 1.23 0.83 4 12 16Historical controls 1.06 0.85 6 10 16Ratio of SDs1.03-folda All patients entering the trial after 1990 were excluded when the protocol’s inclusion criteria were changed.55© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical estimates of bias associated with non-random allocation5020105Odds ratio210.50.20.10.050.02RCTsHCTsFIGURE 12 Comparison of distribution of results of 14,000 historically controlled studies and 14,000 RCTs resampled from 14 regionswithin the ISTTABLE 17 Comparisons of results of RCTs with results of historically controlled non-randomised studies based on 1000 sets ofresampled studies analysed by region (IST)Percentage of studies withstatistically significantresults (p < 0.05):Average OR SD log OR benefit/harmRegion RCT HCT Ratio RCT HCT Ratio RCT HCTSouth Italy 0.56 1.00 1.79 0.29 0.30 1.03 49/0 3/3Scotland 0.78 1.23 1.58 0.35 0.33 0.93 12/0 0/9North Italy 0.94 1.15 1.22 0.29 0.29 0.99 4/2 1/8New Zealand 0.81 0.97 1.20 0.29 0.30 1.03 12/0 4/2Switzerland 0.97 1.07 1.10 0.30 0.30 1.00 3/2 1/4Australia 1.30 1.39 1.07 0.28 0.30 1.07 0/13 0/23Mid-Italy 0.89 0.92 1.03 0.31 0.30 0.98 6/1 5/1The Netherlands 0.88 0.85 0.97 0.28 0.29 1.03 6/1 8/1Northern England 0.95 0.92 0.97 0.36 0.35 0.98 4/2 4/2Spain 1.00 0.93 0.93 0.27 0.28 1.05 2/2 5/1Southern England 1.01 0.87 0.86 0.37 0.36 0.99 2/2 6/1Poland 0.91 0.70 0.77 0.28 0.28 1.01 6/1 27/0Norway 0.78 0.48 0.62 0.28 0.28 0.99 14/1 73/0Sweden 0.94 0.44 0.47 0.32 0.29 0.93 4/2 81/0Total 0.89 0.88 0.99 0.35 0.44 1.23 9/2 16/456Table 18 displays the same results accordingto the region. These results demonstrate abroadly similar pattern to the overall findings:there was no increase in variability of historicallycontrolled studies compared with RCTs and asystematic bias in six of the eight regions infavour of carotid surgery. There was variability inthe average magnitude of the bias by region(Region 1 showing the largest overestimateof harm and Region 8 showing the largestoverestimate of benefit). There was notableheterogeneity in the results of the RCTs between

Health Technology Assessment 2003; Vol. 7: No. 271002020105Odds ratio210.50.20.10.050.020.01RCTsHCTsFIGURE 13 Comparison of distribution of 8000 historically controlled studies and 8000 RCTs resampled from eight regions withinthe ECSTTABLE 18 Comparisons of results of RCTs with results of historically controlled non-randomised studies based 1000 sets of resampledstudies analysed by region (ECST)Percentage of studies withstatistically significantresults (p < 0.05):Average OR SD log OR benefit/harmRegion RCT HCT Ratio RCT HCT Ratio RCT HCTRegion 1 1.53 3.24 2.12 0.52 0.60 1.15 0/14 0/54Region 2 0.78 0.79 1.01 0.57 0.56 0.98 7/1 6/1Region 3 0.71 0.64 0.90 0.58 0.55 0.95 7/0 11/0Region 4 0.52 0.42 0.81 0.68 0.68 1.00 13/0 25/0Region 5 2.68 2.04 0.76 0.72 0.67 0.93 0/24 0/16Region 6 1.20 0.87 0.73 0.54 0.52 0.96 1/4 4/1Region 7 1.28 0.88 0.69 0.59 0.53 0.90 0/6 4/2Region 8 2.85 1.47 0.52 0.57 0.50 0.88 0/47 0/10Total 1.23 1.06 0.86 0.83 0.85 1.02 4/12 6/10these regions, but very similar heterogeneity wasseen between the results of the historicallycontrolled trials.DiscussionOur evaluations have detected the bias in nonrandomisedstudies using either concurrent orhistorical controls. Compared with the use ofrandomised controls, we have shown that use ofnon-randomised controls increases the probabilityof statistically significant study findings (and hencethe likely conclusions), in addition to altering theestimates of treatment effects. The bias observedin the non-randomised studies acted in two ways,which we refer to as systematic and unpredictablecomponents of bias.Systematic biasSystematic bias acts consistently in a givendirection leading to differences in average overall57© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical estimates of bias associated with non-random allocation58results between non-randomised and randomisedstudies. The bias was observed for the ECSThistorically controlled comparisons leading tooverestimates of the benefit of carotid surgery,both for individual regions and when the resultswere aggregated across regions. This pattern isconsistent with the conclusions of Sacks andcolleagues, 27 who noted in their review ofhistorically controlled studies for six medicalinterventions that “biases in patient selectionmay irretrievably weight the outcome ofhistorically controlled studies in favour of newtherapies”.Systematic biases were also noted in some of thehistorically controlled studies in the individualregions in the IST analysis, but here they wereseen to vary in direction and magnitude,sometimes overestimating benefit and sometimesoverestimating harm.Systematic bias in historically controlled studiesarises from there being time trends in theaverage outcomes of participants in a study,regardless of which treatment they receive.Details of the outcomes and characteristicsof the participants in the ECST are presentedin Tables 41 and 42 in Appendix 8. For fiveregions there was a reduction in the adverseevent rate of between 1 and 7% (averaged acrossboth treatment and control) between the trialperiods, whereas for three regions there was anincrease of between 1 and 14%. The changewas statistically significant (p < 0.01) in oneregion.How do such trends arise? There are a limitednumber of options: they must arise throughvariation over time in the case-mix, and henceprognosis, of participants recruited to the trial (asproposed by Sacks and colleagues 27 ), throughdifferences in other healthcare interventions thatthe participants receive or through changingassessments of outcome. These variations maythemselves be haphazard or due to systematicmechanisms (such as changes in patient referraland recruitment or in patient management). Someof these potential causes may be measured, such asbaseline risk factors, but many may go unnoticedand are not assessed.Tables 39 and 42 in Appendix 8 show summariesof the distribution of important baseline riskfactors for IST and ECST, respectively. For bothtrials there were differences in the risk factors ofparticipants between the first and second halves ofthe trial, although the patterns of these differenceswere not consistent between regions, and it is notimmediately obvious how they relate to differencesin outcome. It seems likely that the differencesoccur in part due to unmeasured changes withinthe trials, but that there may also be differentmechanisms causing systematic bias in differentregions.Why should there be a time trend in outcome inthe ECST? Patients were only entered into thetrial when an investigator judged that in the caseof the individual patient there was uncertainty asto whether surgery would be beneficial orharmful. One possibility is therefore thatthroughout the very long recruitment period(12.5 years) investigators joined or left the trialwho had systematically different opinions on whowas suitable for randomisation. Six of the eightregions showed significant reductions (p < 0.05)in the proportion of patients recruited with

Health Technology Assessment 2003; Vol. 7: No. 27Unpredictability in biasWhen bias acts unpredictably, it will sometimeslead to an overestimation and sometimes tounderestimation of an effect. Although thesebiases may on average ‘cancel out’ across a set ofstudies such that no difference is observed inaverage ORs, the biases will still affect the resultsof individual studies. The presence of systematicbias may therefore be missed if the comparison ofresults is restricted to a comparison of averagevalues, as was done in five of the eight previousreviews summarised in Chapter 3. 25–28,32Unpredictable over- and underestimation willincrease the variability (or heterogeneity) of theresults of a set of studies. In the concurrentcomparisons such an increase in variability(measured by the standard deviation) wasobserved for the IST (Table 15), even though theaverage treatment effects in the concurrentlycontrolled and randomised studies were thesame. A similar pattern was observed forhistorically controlled studies generated from theIST when the haphazard within-region timetrends were aggregated in the overall analysis(Table 16).How do these biases occur, and how do they differfrom the variability seen between RCTs? Variabilityalways occurs between the results of multipleRCTs. The principal reason is the ‘play of chance’or sampling variation. A treatment effect observedin a particular RCT is unlikely to be the preciseeffect of the intervention. For example, randomlydividing the study sample into two does notguarantee that the groups are identical in allrespects, and the differences that do exist in casemixwill lead to either under- or overestimates ofthe treatment effect in an individual trial. We donot normally talk about these differences asbiases, but rather as uncertainties. We know thedistribution with which under-and overestimatesarise in RCTs, enabling us to draw correctinferences within specified degrees of certainty. Wecannot identify whether a particular trial isaffected by such bias, but we can calculate boundswithin which we are reasonably sure possible biasis encompassed, which we term confidenceintervals. Importantly, we know that the possibledifferences between the groups due to samplingvariation (and hence confidence intervals) reducewith increasing sample size.The extra variability we see in the nonrandomisedstudies arises in a similar but moretroubling manner. Rather than randomly dividinga single group of individuals, we start with twodifferent groups of individuals. We therefore startwith differences between the groups in measurableand unmeasurable factors. These potentiallyinclude differences in case-mix, additionaltreatments and methods of assessment ofoutcome. Importantly, in addition to not beingable to identify all these differences, we may notknow in which way many of the factors act, so thatthere is overall uncertainty as to whether they willcause under- or overestimates of the treatmenteffect. Sampling from these populationsintroduces the same sampling variation as in theRCT. While we can estimate the impact of thesampling variation (and calculate standardconfidence intervals), there is no mathematicalway of knowing how pre-existing differencesbetween the groups behave. It is therefore notpossible to include an allowance in the confidenceinterval for a single study that accounts for theextra uncertainty introduced through unsystematicbias. As we cannot mathematically allow for thisvariation when drawing conclusions, it isappropriate to call such extra variation ‘bias’ eventhough it is ‘uncertain’. In contrast to samplingvariation, the extra uncertainty is independent ofsample size as it is a feature of the pre-existingdifferences between the two populations fromwhich the samples were drawn.Our resampling studies provide a uniqueopportunity to calculate the distribution of thisextra uncertainty for the specific situations studiedin the IST and ECST by calculating the increase invariance seen with non-randomised concurrentlycontrolled studies compared with RCTs. Thiscomputation is possible as we ensured that foreach study the RCTs are the same size as theconcurrent comparisons, such that the differencesin variability cannot be explained by differences insampling variability. The results of thesecomputations are given in Table 19. The extravariance in log OR was 0.61 for regional ISTcomparisons, 0.57 for UK city IST comparisonsand 0.01 for regional ECST comparisons. Giventhese estimates, it is possible to calculate newadapted confidence intervals for these studies thatallow for these potential uncertain biases inaddition to sampling variation. They areexpressed in Table 19 as multiplicative increases inthe width of the standard confidence intervals. Assampling variability decreases with increasingsample size but the unsystematic bias remainsconstant, the ratio of the extra allowance in thewidth of the confidence interval due tounsystematic bias increases with sample size. Theratios presented in Table 19 reveal that standardconfidence intervals for many non-randomised59© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical estimates of bias associated with non-random allocationTABLE 19 Impact of observed increased variability with sample sizeIST – 14 regions IST – 10 UK cities ECST – 8 regionsObserved ratio of SDs for concurrent controls 2.5 1.8 1.01Increase in variance in log OR attributable tonon-random allocation 0.607 0.570 0.014Total sample sizeMultipliers to confidence interval width to give correct coverage100 1.9 1.5 1.0200 2.5 1.8 1.0500 3.8 2.6 1.11000 5.2 3.5 1.22000 7.3 4.8 1.35000 12 7.5 1.710000 16 11 2.220000 23 15 2.950000 36 24 4.460studies may be an order of magnitude too narrowto describe correctly the true uncertainty in theirresults, but that there are differences in theadjustments that are needed in differentsituations. For example, the confidence intervalcalculated from a concurrently controlled study of1000 participants may be five times too narrow todescribe the true uncertainty for regional IST-typecomparisons, three times too narrow to describethe true uncertainty in UK city IST-typecomparisons, but only 20% too narrow forregional ECST-type comparisons. For sample sizesof 10,000 the confidence intervals are estimated tobe more than 10 times too narrow for the ISTsituations and half the width needed for theECST situation. Of course, in practice onewould not know to what extent the standardconfidence interval under-represented the trueuncertainty.Generalisability and limitations of thefindingsThe value of these findings and estimates dependson the generalisability of the results obtained fromthe IST and ECST and the degree to which theslightly artificial methodology and samples used inthese evaluations are representative of the realityof non-randomised studies.GeneralisabilityThe IST and ECST were chosen for thisinvestigation as (a) they were large trials, (b) theyhad an outcome which was not rare, (c) they weremulticentre trials and (d) the trialists were willingto provide reduced and anonymised data setssuitable for our analyses. Other than the fact thatboth trials relate to stroke medicine, the trialsdiffer considerably. One is a trial ofpharmacological agents (aspirin and heparin)whereas the other is a trial of a surgical procedure(carotid endarterectomy). The treatment in one isacute, being given immediately after the patientshave suffered a severe stroke, whereas in the otherit is preventive, being given to high-risk patients.It is difficult to argue that these trials can beregarded as representative and therefore that theresults are generalisable. However, their resultsshould be regarded as being indicative of thebiases associated with the use of non-randomcontrols. Ideally these resampling study methodsshould be repeated in more trials. In the caseof this project, the time required to generatethe resampling studies and the difficulty inobtaining data sets from multicentre clinicaltrials prevented additional evaluations beingundertaken.It is important also to consider whether the timetrend observed in the ECST is likely to be typicalof those that may be observed in other areas ofhealthcare – especially as it is in agreement withthe trends observed by Sacks and colleagues intheir review across six clinical contexts. 27 Thetrend is one of patient outcomes improving overtime. It is consistent with a general pattern ofaverage outcomes improving with progress inmedical care, which may apply across all medicalspecialities. However, this argument assumes thatthe case-mix of patients being treated is stable,which may not be the case. In some circumstanceschanges in case-mix over time, for good reason,may lead to apparent increases in adverseoutcomes. For example, if medical informationleads to knowledge that the treatment is not suitedto patients at low risk, then a change to excludinglower risk patients from receiving that treatment

Health Technology Assessment 2003; Vol. 7: No. 27may lead to increases in average event rates.Historically controlled studies undertaken in sucha situation may be prone to underestimating thebenefits of treatment and may even falselyconclude that treatment does more harm thangood.The lack of systematic bias with the use ofgeographical controls is based on the presumptionthat geographical differences act in a haphazardmanner, and are as likely to lead to overestimatesof treatment effects as to underestimates. Therandom manner in which concurrent controlgroups were selected in the resampling exerciseensured that across a large number of studiesthese differences would be seen to balance eachother out, albeit possibly increasingunpredictability. This result does not indicate thatgeographically controlled studies are unbiased. Inreality, a single comparison between two areas islikely to be biased, as are meta-analyses ofseveral studies, although the direction in whichthe bias acts may be unknown. In addition,if an investigator chose a geographical controlgroup with knowledge of the likely differencesin case-mix, it would be possible for theselection to be manipulated (consciously orsubconsciously) in such a way as to introducea particular bias, akin to the bias observed inRCTs when treatment allocation is notconcealed. 20Similarly, we should consider whether themechanisms leading to unpredictability in bias,especially in studies generated from the IST, arelikely to apply widely across different clinical areas.Tables 38 and 40 in Appendix 8 show that the casemixof patients recruited to the IST variedbetween locations, both internationally andbetween cities in the UK. These haphazarddifferences, together with differences in otherunknown risk factors and aspects of patientmanagement and outcome assessment, will havecaused the unpredictability in the bias that wasobserved. Evidence is available in all areas ofmedicine that such differences exist, and thereforeit seems reasonable to conclude that theunpredictable behaviour in biases will be observedelsewhere, although the degree of unpredictabilitymay vary.Limitations of the resampling methodologyThe resampling method used participantsrecruited to a randomised controlled trial togenerate non-randomised studies. This, of course,is not what happens in reality, but there arereasons to believe that our approach is more likelyto have led to underestimates than overestimatesof bias.The degree of bias in a non-randomised studydepends on the similarity of the two groups fromwhich treated participants and controls are drawn.Sampling these groups from the same randomisedtrial is likely to have reduced such differences forthe following reasons:1. All participants included in the RCT will havebeen judged to have been suitable for eithertreatment. In a non-randomised studyparticipants who are suitable for only one ofthe two treatments may have been recruited tothat arm: there is usually no formal assessmentthat they would have been considered suitablefor the alternative. This difference will nearlyalways act to increase differences in outcomebetween the groups.2. The RCT was conducted according to aprotocol, describing methods for recruiting,assessing, treating and evaluating the patients.This will have reduced the variability withinthe trial. Although some non-randomisedstudies are organised using a protocol, manyare not.3. All participants included in the trial wererecruited prospectively. In non-randomisedstudies, especially those using historicalcontrols, participants are likely to have been‘retrospectively’ included in the study,potentially introducing additional bias.On balance, it could be argued that usingrandomly chosen international comparisons forselection of concurrent controls may be regardedas rather artificial and likely to have increaseddifferences between groups. In reality, aconcurrent control group in a non-randomisedstudy would be selected to minimise likelydifferences between groups, and such longdistancegeographical comparisons wouldprobably be avoided. It is perhaps more realistic tofocus on the magnitude of the biases observed inthe concurrent comparisons generated from theUK cities in the IST as being more representativeof what might occur in reality. The unsystematicbias seen here was less than that observed ininternational comparisons, but still large enoughto lead many studies falsely to obtain significantfindings of both benefit and harm.Importantly, we have concentrated on only oneaspect of quality in non-randomised studies: thereare other biases to which they are susceptible inthe same way as are RCTs.61© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical estimates of bias associated with non-random allocationConclusionsIndividual non-randomised studies may haveseriously biased results when either historical orconcurrent controls are used. Biases may belarge enough to mask both small and moderatetreatment effects, to the extent that nonrandomisedstudies may observe statisticallysignificant effects acting in the wrongdirection.While the use of historical controls may frequentlylead to overestimates of effectiveness of theexperimental treatment, this depends on theunderlying time trend of improving outcomes inthe patients being studied. Such a trend may notalways apply, however, especially when the casemixof those being considered for treatment alsochanges over time.Geographical variations in the provision of careand in the case-mix of patients being consideredfor treatment will lead to bias in studies usinggeographical concurrent controls. Differences incase-mix between treatment and control groupsare likely to be haphazard, such that the size andmagnitude of the biases may be unpredictable.The magnitude and nature of biases may differconsiderably between clinical situations.While results of RCTs and non-randomised studiesmay appear to agree on average across manystudies, this does not indicate that non-randomisedstudies are reliable, as individually they are affectedby additional unpredictability. Failure to recognisethis aspect may have led previous reviews falsely tounderestimate the bias in non-randomised studieswhen they compared average results.62

Health Technology Assessment 2003; Vol. 7: No. 27Chapter 7Empirical evaluation of the ability of case-mixadjustment methodologies to control forselection biasIntroductionCase-mix (or risk) adjustment methods are usedwidely throughout healthcare research to enablecomparisons to be made between groups that arenot directly comparable owing to differences inprognostic (or confounding) factors. For example,comparisons between outcomes from differenthospitals are often confounded by the severity ofthe patients that they treat, and it is recommendedthat outcomes should be compared only whendifferences in case-mix (severity) are adjustedfor. 136,137 The philosophy of case-mix adjustmentis to ‘level the playing field’ during analysis,enabling a comparison of like-with-like to be madeat the point of analysis, even if comparable groupscould not be generated by the study design. Casemixadjustment is an attempt to achieve byanalysis what could not be done (or was not done)by the design.Consideration of case-mix is routinelyrecommended in the analysis of non-randomisedstudies. Guides for assessing the validity of nonrandomisedstudies for both therapy and harmrecommend readers to assess first whether‘investigators demonstrate similarity in all knowndeterminants of outcome’ and, if not, whetherthey ‘adjust for differences in analysis’. 138,139 Manyepidemiological and biostatistical texts advocateadjustment for differences in baseline covariateswhen they are observed, both in non-randomisedcontrolled studies and even in RCTs. 140–142We will focus on comparisons between two groups,which we refer to as treatment (or experimental)and control. In principle, many of the methodsdiscussed extend to multiple armed trials. Fourapproaches to dealing with differences in case-mixin non-randomised studies are commonlyencountered in the medical literature:1. Comparison of baseline characteristics.The baseline clinical and demographiccharacteristics of the two groups are comparedto ascertain whether differences in case-mixexist and hence determine the certainty withwhich the observed difference can be attributedto the intervention and not to confoundingfactors. This method does not attempt to adjustfor differences in case-mix but simply todiscover whether there is evidence ofconfounding and make inferences accordingly.The presence of baseline differences is usuallydetermined by tests of statistical significance,although such tests do not relate directly tocomparability. 1432. Standardised or stratified analyses. Studyparticipants are divided into groups (strata)with the same characteristics. Stratified analyseswork by making ‘like-with-like’ comparisonswithin each of these groups. The overalltreatment effect is calculated by computing aweighted average of the within-strata estimatesof the treatment effect. The most popularmethod is that of Mantel and Haenszel. 144Stratification is best used when there are onlyone or two baseline characteristics, and isfrequently used in epidemiological research tostandardise for differences in age and/or sex.3. Multiple regression. Regression models (linearregression if the outcome is continuous,logistic regression if it is binary and Coxmodels if censoring occurs) estimate how eachprognostic factor relates to outcome. When thecomparison between the groups is made,adjustments are added to or subtracted fromthe estimated treatment effect to account forthe impact of differences in each of thebaseline covariates according to their estimatedrelationship with outcome. The two stages ofthe process happen simultaneously, so that theresults depend on the correlations betweenbaseline characteristics and treatmentallocation. Stepwise regression procedures arecommonly used to identify covariates that aresignificantly related to outcome, and work in asequential manner such that adjustments areonly made for the subset of covariates thoughtto matter.4. Propensity score methods. Propensity scoresare the least familiar method. Whereas63© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection bias64regression models described in (3) model howcovariates relate to outcome, propensity scoremethods model how the same covariates relateto treatment allocation. 145–147 The principle isthat the propensity score summarises themanner in which baseline characteristics areassociated with treatment allocation, so thatselection bias is removed when comparisonsare made between groups with similarpropensity scores. 146 The method involvescalculation for each individual in the datasetthe propensity probability that estimates theirchance of receiving the experimentalintervention from their baseline characteristics.In an RCT, where there should be norelationship between baseline characteristicsand treatment assignment, this probability willbe the same for each participant (e.g. 0.5 if thegroups are of equal size). In non-randomisedstudies, it is likely that treatment assignmentdoes depend on baseline covariates, and thatpropensity scores will vary betweenindividuals. For example, patients with moresevere disease may be more likely to receiveone treatment than the other. Propensityscores in such a situation would relate todisease severity, and the average propensityscore in the experimental group will differfrom the average in the control group.Estimation of propensity scores is typicallyundertaken using multiple logistic regressionmodels, the outcome variable being treatmentallocation.What evidence is there that these methods actuallyadjust for selection bias in non-randomisedstudies? Although there is plenty of literaturedemonstrating that case-mix adjustment canchange estimates of treatment effects, none of thetexts that we have consulted cite any empiricalevidence demonstrating that case-mix adjustment,on average, reduces bias. Broader searches of themethodological literature related to case-mixlikewise did not identify supporting empiricalevidence.To the contrary, there are hints in the literaturethat case-mix adjustment methods may notadequately perform the task to which they areapplied. Of the eight reviews in Chapter 3, twocompared adjusted and unadjusted estimates oftreatment effects from non-randomised studieswith the results of similar RCTs, and noted thatthere was little evidence that adjustmentconsistently moved the estimates of treatmenteffects from non-randomised studies towards thoseof RCTs. 25,27In an attempt to provide empirical evidence ofthe value of case-mix adjustment, we have usedthe same non-randomised studies generated byresampling participants from the IST 132 andECST 133 data sets (described in Chapter 6) toevaluate the performance of eight different casemixadjustment methods in controlling forselection bias. As explained previously,resampling of data from the trials was used togenerate randomised and non-randomisedstudies of different designs in such a way that thedifferences between their results with respect tolocation and spread could only be attributed toselection bias. Case-mix adjustment methodswere then applied to each resampled nonrandomisedstudy making use of the availablebaseline data (different in the two trials – seeTable 20). The distribution of results of theadjusted non-randomised studies was thencompared with both the distribution of unadjustedresults and the distribution of the results of theRCTs, to see whether there was any evidence thatadjustment had reduced or removed the selectionbias.In our investigations we evaluate the ability ofcase-mix adjustment to control for three differenttypes of bias:1. Naturally occurring systematic bias, whereresults are consistently either all overestimatesor all underestimates of the treatment owing tosome naturally occurring unknown allocationmechanism (as was observed in the ECSThistorical comparisons in Chapter 6).2. Naturally occurring unpredictable bias, whereresults are biased variably with differentmagnitudes in different directions leading toboth underestimates and overestimates oftreatment effects (as was observed in the ISTconcurrent comparisons in Chapter 6).3. Bias arising where allocation to treatment isprobabilistically linked to a prognostic variable,such that participants are more likely or lesslikely to receive treatment according to theirobserved characteristics. This in fact is a modelof practice in much of medicine, wheretreatment decisions are made according to theconditions and characteristics that each patientdisplays (allocation by indication). Although wehave no realistic way of mimicking such amechanism for the two trial data sets, wegenerate two artificial scenarios when allocationrelated (a) to the value of a single prognosticcovariate, and, more realistically, (b) to anunknown function of a set of prognosticcovariates.

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 20 Baseline covariates used in case-mix adjustment models for the IST and ECSTISTBinary covariatesSex (male/female)Symptoms noted on wakingConsciousnessAtrial fibrillationContinuous covariatesAgeDelay to presentationSystolic blood pressureECSTSex (male/female)Residual neurological signsPrevious MIAnginaCurrent prophylactic aspirin useAgeDegree of stenosisUnordered categorical variablesInfarct visible on CT scanType of strokeOrdered categorical variablesNeurological deficit score (7 categories)Presenting stroke (4 categories)The investigative method provides an opportunityto make comparisons between different case-mixadjustment strategies: matching baseline groups,stratification, regression and propensity scoremethods.MethodsGeneration of samplesThe principles of our resampling methodologywere discussed in detail in Chapter 6. Studies offixed sample sizes with randomised and nonrandomiseddesigns (described below) weregenerated for each region in each of the IST andECST data sets by selectively samplingparticipants, the whole process being repeated1000 times. In each trial baseline data onimportant prognostic variables had been recordedfor each participant at the point of recruitment.These variables (we will refer to them ascovariates) were used in the analyses to adjust fordifferences in case-mix. Details of the covariatesavailable for each study are given in Table 20 andAppendix 8.Samples with ‘naturally’ occurring biasesHistorically controlled and concurrently (nonrandomised)controlled studies were generatedfrom the IST and ECST data sets as described inChapter 6. We thus obtained results for:1. 14,000 historically controlled studies basedon the 14 international regions from the ISTand 14,000 corresponding RCTs, all of samplesize 200 (100 per arm)2. 14,000 concurrently controlled studies basedon the 14 international regions from the ISTand 14,000 corresponding RCTs, all of samplesize 200 (100 per arm)3. 10,000 concurrently controlled studies basedon the 10 UK cities within the IST and 10,000corresponding RCTs, all of sample size 200(100 per arm)4. 8000 historically controlled studies based onthe eight international regions from the ECSTand 8000 corresponding RCTs, all of samplesize 80 (40 per arm)5. 8000 concurrently controlled studies based onthe eight international regions from the ECSTand 8000 corresponding RCTs, all of samplesize 80 (40 per arm).Different case-mix adjustment methods wereapplied individually to each of these 54,000 nonrandomisedstudies, and the results werecompared with the results from the corresponding54,000 RCTs.Samples with bias related to ‘known’ differencesin case-mix (allocation by indication)In addition to the standard historically andconcurrently controlled designs, for the purposesof evaluating the performance of case-mixadjustment we have included two further designsin which bias relates to known relationships withprognostic variables. This has been done for tworeasons. First, we wished to evaluate how well casemixmethods work in situations when we havedirect knowledge of the bias-inducing mechanismthat they are trying to correct. Second, we wishedto mimic crudely clinical database-type studies, in65© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection biasRegion Trial allocation Number of neurological deficits0 1 2 3 4 56Northern ItalyAspirinAvoid aspirinT 0,1C 0,1T 1,1C 1,1T 2,1C 2,1T 3,1C 3,1T 4,1C 4,1T 5,1C 5,1T 6,1C 6,1Central ItalyAspirinAvoid aspirinT 0,2C 0,2T 1,2C 1,2T 2,2C 2,2T 3,2C 3,2T 4,2C 4,2T 5,2C 5,2T 6,2C 6,2Southern ItalyAspirinAvoid aspirinT 0,3C 0,3T 1,3C 1,3T 2,3C 2,3T 3,3C 3,3T 4,3C 4,3T 5,3C 5,3T 6,3C 6,3ScotlandAspirinAvoid aspirinT 0,4C 0,4T 1,4C 1,4T 2,4C 2,4T 3,4C 3,4T 4,4C 4,4T 5,4C 5,4T 6,4C 6,4Northern England and WalesAspirinAvoid aspirinT 0.5C 0,5T 1.5C 1.5T 2.5C 2.5T 3.5C 3.5T 4.5C 4.5T 5.5C 5.5T 6.5C 6.5Southern EnglandAspirinAvoid aspirinT 0,6C 0,6T 1,6C 1,6T 2,6C 2,6T 3,6C 3,6T 4,6C 4,6T 5,6C 5,6T 6,6C 6,6AustraliaAspirinAvoid aspirinT 0,7C 0,7T 1,7C 1,7T 2,7C 2,7T 3,7C 3,7T 4,7C 4,7T 5,7C 5,7T 6,7C 6,7The NetherlandsAspirinAvoid aspirinT 0,8C 0,8T 1,8C 1,8T 2,8C 2,8T 3,8C 3,8T 4,8C 4,8T 5,8C 5,8T 6,8C 6,8New ZealandAspirinAvoid aspirinT 0,9C 0,9T 1,9C 1,9T 2,9C 2,9T 3,9C 3,9T 4,9C 4,9T 5,9C 5,9T 6,9C 6,9NorwayAspirinAvoid aspirinT 0,10C 0,10T 1,10C 1,10T 2,10C 2,10T 3,10C 3,10T 4,10C 4,10T 5,10C 5,10T 6,10C 6,10PolandAspirinAvoid aspirinT 0,11C 0,11T 1,11C 1,11T 2,11C 2,11T 3,11C 3,11T 4,11C 4,11T 5,11C 5,11T 6,11C 6,11SpainAspirinAvoid aspirinT 0,12C 0,12T 1,12C 1,12T 2,12C 2,12T 3,12C 3,12T 4,12C 4,12T 5,12C 5,12T 6,12C 6,12SwedenAspirinAvoid aspirinT 0,13C 0,13T 1,13C 1,13T 2,13C 2,13T 3,13C 3,13T 4,13C 4,13T 5,13C 5,13T 6,13C 6,13SwitzerlandAspirinAvoid aspirinT 0,14C 0,14T 1,14C 1,14T 2,14C 2,14T 3,14C 3,14T 4,14C 4,14T 5,14C 5,14T 6,14C 6,1466FIGURE 14 Resampling structure for IST based on number of neurological deficits

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 21 Sampling probabilities use to generate non-randomised designs mimicking ‘allocation by indication’(a) Sampling probabilities by number of neurological deficits, IST0 1 2 3 4 5 6Aspirin 1.0 0.9 0.8 0.6 0.4 0.2 0.1Avoid aspirin 0.1 0.2 0.4 0.6 0.8 0.9 1.0(b) Sampling probabilities by consciousness on presentation, ISTFully alertDrowsy or unconsciousAspirin 1.0 0.5Avoid aspirin 0.5 1.0(c) Sampling probabilities by outcome, ISTAdverse outcomeNo adverse outcomeExperimental 0.75 1.00Control 1.00 0.75which patients have been allocated interventionsbased on their characteristics. These additionalanalyses were carried out on the IST data only.Non-randomised studies mimicking ‘allocation byindication’ were generated by identifying strongprognostic covariates in the IST and biasingallocation according to values of these covariates.These studies were based on a simplistic form ofallocation by indication, in which the decision totreat depends on a single observed covariate. Twodifferent covariates were considered separately:the number of neurological deficits and the levelof consciousness at admission. The prognosticvalue of these covariates is summarised inAppendix 8.To generate bias, participants within each regionwere divided into groups according to their valuesof the covariate and the treatment that theyreceived. For example, Figure 14 shows thecategorisation of participants in the IST accordingto whether they receive aspirin (T) or avoid aspirin(C), the number of neurological deficits with whichthey present (0–6, first subscript) and the regionwithin which they were recruited (1–14, secondsubscript). Participants were resampled from thisstructure with sampling probabilities determinedaccording to their allocation and covariate value.For both covariates sampling probabilities werechosen such that the treated group was weightedtowards participants with good prognosis, whereasthe control group was weighted towardsparticipants with poor prognosis, leading to anoverestimate of the benefit of experimentaltreatment. The sampling probabilities used withthe two covariates are given in Table 21(a) and (b).The same sampling probabilities were used in allregions.In practice, allocation by indication is morecomplicated than the single covariate allocationdescribed above – it depends on some unknowncombination of multiple pieces of covariateinformation that are predictive of prognosis, andis likely to vary both between individual cliniciansand over time. We wished to include studies withsuch a complex bias in our evaluation toinvestigate how well case-mix adjustment methodsmight work in more typical database-type analyses.Research into medical decision making hascharacterised some decision processes usingregression techniques to provide modelsexplaining how covariate information is combinedto make treatment decisions. Although it would bepossible to generate biased allocations using sucha decision model, we did not use this approach asit is too simplistic and artificial to test properly theability of case-mix adjustment methods to adjustfor selection bias. Such analyses wouldoverestimate the ability of case-mix adjustmentmethods to adjust for bias due to the circularity ofone regression model (the case-mix adjustmentmethod) being used to estimate parameters of anunderlying regression model (the allocationprocess). The many treatment decisions that didnot fit with the underlying regression model wouldnot be accounted for.The second approach that we used is based on theobservation that when decisions are made inrelation to prognosis, those decisions naturally67© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection bias68relate to patient outcome. To generate a biasrelated to prognosis, therefore, we can takeadvantage of the fact that the outcomes in thesepatients are already known, thus skipping theprocess of selecting a prognostic model to be usedto represent treatment decisions. We havetherefore generated a prognostic bias mimicking‘allocation by indication’ by stratifying patients ineach region according to outcome and treatment,and differentially sampling patients with good andbad outcomes in the two treatment arms. Thesampling probabilities used are given in Table 21(c).Probabilities were again chosen to lead tooverestimation of treatment benefit, and the samesampling probabilities were used in all samples.Correcting for this selection process is likely to bemore testing than correcting for real allocation byindication as it is based on information notdirectly available to the treatment allocator, whowill only have available signs, symptoms, historyand results of investigations on which to basetreatment.The resampling methods were repeated 1000times as for the historically and concurrentlycontrolled studies (Chapter 6). Thus, for the IST14,000 non-randomised studies (1000 for eachregion) were generated with bias related toneurological deficit score, 14,000 with bias relatedto level of consciousness and 14,000 with biasrelated to outcome, all of sample size 200. Theresults of these studies were compared with thoseof 14,000 RCTs of the same sample size sampledfrom the same data. The case-mix adjustmentmethods were applied to each of these 42,000non-randomised studies.Case-mix adjustment methodsEight case-mix adjustment strategies wereinvestigated: matching baseline groups,stratification, three variants of regression modelsand three propensity score methods.Exclusion of non-matching groupsThe first analysis investigated the degree to whichthe absence of differences in case-mix was relatedto the absence of selection bias. The significanceof baseline differences for the 10 prognosticvariables for IST and eight prognostic variablesfor ECST that are listed in Table 20 was testedusing t-tests (continuous variables), chi-squaredtests (binary and unordered categorical variables),and chi-squared tests for trend (orderedcategorical variables) using the STATA commandsttest, tabulate and mhodds, respectively.Studies were then classified by the number ofcovariates with significant (p < 0.05) baselinedifferences. Although we do not imagine thatresearchers would desist from analysing orpublishing results of non-randomised studies withsignificant differences in one or more baselinecovariates, this analysis allows us to investigate theadvice given to readers to assess whether‘investigators demonstrate similarity in all knowndeterminants of outcome’. 138,139Stratification on a single factorThe Mantel–Haenszel stratified method 144 wasapplied to all non-randomised studies for bothtrials. Stratification was undertaken according tothe most strongly prognostic factor for each of thetwo trial data sets: the neurological deficit scorewas used for IST (seven categories) and the degreeof stenosis for ECST (four categories). The STATAmhodds command was used to obtain an overallOR together with 95% confidence interval(calculated using the standard equation proposedby Robins and colleagues 148 ).Multiple logistic regression modelsMultiple logistic regression models were fitted tothe data from each non-randomised study for bothtrials. Three variants of the method were applied.The first included all prognostic variables listed inTable 20, and is termed the ‘full model’. For boththe IST and ECST studies a variable was alsoincluded indicating treatment group, the value ofwhich yielded the estimate of log OR (and 95%confidence interval) of the adjusted treatmenteffect. Analysis was done using the STATA logitcommand.Two stepwise alternatives to the full model wereconsidered, both using the STATA sw logitcommand. Both used backward selectiontechniques removing non-significant variablesfrom the full model one at a time. The modelsdiffered in the significance level used to decidewhether to remove a variable. The first methodremoved variables with p-values >0.15. Thesecond method used a stricter p-to-remove valueof 0.05, requiring greater evidence of arelationship between a covariate and outcome forthe covariate to be included in the final model. Inboth situations the variable for treatment groupwas forced to remain in the model throughout thestepwise procedure, its value again yielding theestimate of log OR (and 95% confidence interval)for the adjusted treatment effect.The logistic regression models estimated only themain effects associated with the covariates in themodel (as is typical of logistic regression analysespublished in the medical literature). No attempt

Health Technology Assessment 2003; Vol. 7: No. 27was made to investigate interactions, or toinvestigate non-linear models of continuouscovariates. Although these could be regarded asimportant oversimplifications, the size of the dataset (n = 200 for IST, n = 80 for ECST) means thatsuch investigations would be underpowered.Propensity score methodsAs noted above, an individual’s propensity score istheir probability of being in the treatment groupgiven their baseline data. Once the score has beencalculated for all individuals in the data set, it canbe used in the analysis in one of three ways. First,participants in the experimental and controlgroups can be selected to generate groups withsimilar distributions of propensity score bymatching each individual in the experimentalgroup with an individual in the control group whoshares the same (or a very similar) propensityscore. Participants for whom no match can befound are excluded from the analysis. Second, astratified analysis can be undertaken, theparticipants being divided into, say, quintilesaccording to their propensity score, andcomparisons made within each of the five groups.As with conventional stratified analysis, the overalltreatment effect is estimated by calculating aweighted average of the within strata estimates ofthe treatment effect. Third, the propensity scorecan be used as a predictor of outcome in a logisticregression model. The model will estimate therelationship between propensity score andoutcome, and make a suitable adjustment to theestimate of treatment effect according to thedifference in mean propensity score betweentreated and control groups.The first stage of the propensity score methodinvolved calculating propensity probabilities for allmembers of each data set, using a logisticregression model with ‘treatment’ as the outcomevariable. The standard sets of 10 covariates for theIST and eight covariates for the ECST (as listed inTable 20) were included in this model, with nointeraction terms being considered. Thelogistic command in STATA was used to fit themodel and obtain estimated propensity scores foreach participant. The propensity score was thenused in the three different ways: matching,stratification and regression.Matching on propensity scores involved pairingeach treated participant with a control participantwho has an identical or very similar propensityscore. Our definition of ‘very similar’ wasRosenbaum and Rubin’s suggestion of beingwithin 0.25 SD of the logit of the propensityscore. 145 This approach takes into account theobserved variability of the computed propensityscores in each study. The difference d in logitpropensity scores was calculated for all possiblepairs of treated and control participants. Thepairings were sorted in ascending order accordingto d, and pairs where d was greater than ourmatching criterion were dropped. Then the firstmatched pair (t 1 , c 1 ) was selected, and all otherpairs including either of t 1 and c 1 dropped. Theprocess was repeated through the data set,selecting the next available matched pair (t i , c i ),and dropping all other pairs including either of t ior c i . Participants in both groups who were notmatched were excluded from the analysis. Theindividual pairing was not exploited in theanalysis: the OR was computed as the ratio of theodds of the outcome in the treated group to theodds in the control group in the standard way,with a standard error for log OR calculated usingthe large sample estimate.The stratified analysis used the Mantel–Haenszelmethod to estimate a common OR across theparticipants grouped according to their propensityscore. In each analysis five groups were used,defined by calculating the quintiles of theobserved propensity scores (pooled acrosstreatment and control groups). Calculations wereperformed using the STATA mhodds command.The regression analysis estimated the treatmenteffect adjusted for differences in propensity scoreby fitting a logistic regression model to theoutcome data with just two covariates, thetreatment group and the log OR of the propensityscore. The model was fitted using the STATAlogistic command.Statistical analysesAs with the analysis of bias related to study designs(Chapter 6), the focus of the statistical analysis wason describing and comparing the location(average) and spread (variability) of the treatmenteffects. Distributions were considered first for theRCTs and the non-randomised studies withoutadjustment, and second for the non-randomisedstudies with each of the eight case-mix strategiesdescribed above. In all analyses treatment effectswere expressed as log OR with 95% confidenceintervals, and interpreted according to theirstatistical significance assessed at the 5% levelusing a two-sided test.Distributions of results were consideredgraphically using dotplots, and through statisticssummarising location [the ‘average OR’, computed69© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

70TABLE 22 Comparison of concurrently and historically controlled studies with results of RCTs resampled from 14 regions within the IST analysed according to the number of significant differencesin baseline covariatesStudy design Maximum no. of significant Percentage of Average Ratio of Percentage of studies with statisticallybaseline differences studies meeting OR SD of log OR significant results (p < 0.05)(p < 0.05) for match a criterionBenefit Harm TotalRCT 0.89 9 2 11Non-randomised historical control Any number 100 0.88 1.3 15 4 190 or 1 or 2 79 0.89 1.3 14 4 180 or 1 55 0.89 1.2 12 3 150 23 0.92 1.2 12 3 15RCT 0.91 7 2 9Non-randomised concurrent control Any number 100 0.91 2.6 29 22 510 or 1 or 2 20 0.93 2.8 27 19 460 or 1 6 0.91 2.1 26 16 420 1 0.86 1.5 20 7 27a Baseline differences were considered for 10 variables: time since symptoms, consciousness, age, sex, presence of atrial fibrillation, stroke noticed on waking, systolic bloodpressure, infarct visible on CT scan, type of stroke and a score of eight neurological deficits.TABLE 23 Comparison of concurrently and historically controlled studies with results of RCTs resampled from eight regions within the ECST analysed according to the number of significantdifferences in baseline covariatesStudy design Maximum no. of significant Percentage of Average Ratio of Percentage of studies with statisticallybaseline differences studies meeting OR SD of log OR significant results (p < 0.05)(p < 0.05) for match a criterionBenefit Harm TotalRCT 1.23 4 12 16Non-randomised historical control Any number 100 1.06 1.0 6 10 160 or 1 or 2 86 1.01 1.0 7 9 160 or 1 57 0.93 1.0 8 7 150 18 0.81 0.9 9 3 12RCT 1.08 3 5 9Non-randomised concurrent control Any number 100 1.08 1.0 3 6 90 or 1 or 2 69 1.06 1.0 3 5 80 or 1 40 1.04 1.0 3 4 70 12 1.17 0.9 3 4 7Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection biasa Baseline differences were considered for eight variables: severity of presenting stroke, degree of stenosis, age, prophylactic aspirin use, angina, previous myocardial infarction,residual neurological signs and sex.

Health Technology Assessment 2003; Vol. 7: No. 27as the exponential of the mean of the log OR] andspread [the standard deviation of the observed logOR]. As in Chapter 6, ratios of the the averageORs of the RCTs and the average ORs of the nonrandomisedstudies (adjusted or unadjusted)quantify systematic bias, while ratios of their SDsindicate unpredictability in the bias. The likelyconclusions of the analyses were investigated byconsidering the percentage of studies for eachanalysis which reported statistically significant (p< 0.05) results, separately in the direction of harmand of benefit.ResultsExclusion of non-matching groupsTable 22 presents the results for analyses of the ISTaccording to the number of covariates thatdiffered significantly between treatment andcontrol groups. Table 23 presents comparableanalyses for the ECST.For the IST, 23% of the historically controlledstudies appeared to have comparable groups inthat they had no statistically significant differencesin baseline covariates (Table 22). There did appearto be a reduction in both systematic andunpredicatable dimensions of bias as the numberof differing covariates reduced. However, amongstudies with no significantly unbalanced covariatesthe results were still more variable than those fromthe corresponding RCTs, and there was still anexcess of statistically significant results.Table 23 shows that 18% of the historicallycontrolled non-randomised studies generatedfrom the ECST had no statistically significantdifferences in baseline covariates. A reverse trendwas noted with study results becoming morebiased with fewer significant differences inbaseline, contrary to the hypothesis thatcomparability at baseline predicts reliable results.Only 1% of concurrently controlled studies fromthe IST had no significant differences at baseline(Table 22), and reductions in excess variability wereless marked than for historically controlled studies.Further, the absence of statistically significantdifferences did not guarantee comparability, withspuriously statistically significant treatment effectsstill being common when treated and controlgroups appeared to match.As shown in Table 23, 12% of the concurrentlycontrolled non-randomised studies generatedfrom the ECST had no statistically significantdifferences in baseline covariates, but againsystematic bias was actually higher in thissubgroup of apparently comparable studies thanin the studies which had significant differences inat least one baseline covariates.Adjustment for ‘naturally’ occurringbiasesSystematic bias originating from historicallycontrolled studiesThe clearest example of systematic bias wasobserved in historically controlled studies in theECST data, where the average OR estimate was1.06 compared with 1.23 in the RCTs, grosslyunderestimating the harmfulness of treatment(see Chapter 6). The seven case-mix adjustmentmethods were applied to each of the historicallycontrolled studies to investigate the degree towhich the case-mix methods could adjust for thisbias. The results are presented in Table 24 andFigure 15. The adjusted results from six of theseven methods were on average more biased thanthe unadjusted results: only the full logisticregression model appeared to reduce bias. Allmethods also increased variability(unpredictability in bias), the greatest increasebeing with the full logistic regression. Thisincrease in the width of distributions of results isdiscernible in Figure 15.The use of adjustment methods inflates thestandard error of the estimate of a treatmenteffect. The use of propensity score matching led inaddition to a reduction in sample size (as 45% ofparticipants were discarded), further reducingpower. Thus, although many of the adjustedestimates were on average more biased than theunadjusted estimates, only logistic regressionmethods had markedly increased spurioussignificance rates.Systematic bias was also observed in thehistorically controlled studies generated from theIST, although it differed between regions. Overalla small systematic bias was noted in theaggregated results (Table 25). Logistic regressionmodelling showed a similar pattern of behaviouras for the ECST comparison, increasing bothaverage bias and the variability of results.Propensity score methods slightly overadjusted forthe bias, but gave results closest to the RCTresults. Spurious statistical significance ratesdecreased slightly with logistic regression and werecompletely removed by propensity score methods.The case-mix adjusted results for the historicallycontrolled studies from the IST analysis are71© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection biasTABLE 24 Comparison of methods of case-mix adjustment applied to results of historically a controlled studies with results of RCTsresampled from eight regions within the ECSTPercentage of studies withstatistically significantAverageVariability of results results (p < 0.05)OR SD of log OR Ratio with RCT Benefit Harm TotalRCTs 1.23 0.83 4 12 16Historically controlled studiesUnadjusted 1.06 0.85 1.03 6 10 16Stratification 0.99 0.93 1.12 8 10 18Logistic regressionFull model b 1.17 1.96 2.37 12 14 26Stepwise p r = 0.05 c 1.04 1.16 1.40 11 14 25Stepwise p r = 0.15 d 1.01 1.32 1.59 13 15 27Propensity scoreMatched e 0.96 1.09 1.32 5 10 16Stratified 1.01 1.12 1.35 12 10 22Regression 1.00 1.11 1.34 12 10 21a Data excluded after protocol change in 1990.b Full model includes eight covariates.c Mean number of covariates included: 2.3.d Mean number of covariates included: 3.5.e Mean number of patients matched: 44/80.Odds ratio10005002001005020105210.50.20.10.050.020.010.0050.0020.001RCTs HCTs LR(F) LR(5%) LR(15%) MH PS(M) PS(S) PS(R)FIGURE 15 Comparison of methods of case-mix adjustment applied to results of historically controlled studies resampled from eightregions within the ECST. RCT: results from corresponding randomised controlled trials; HCTs: unadjusted historical controls withoutadjustment, LR(F): adjustment using full logistic regression analysis; LR(5%): adjustment with stepwise logistic regression withp r = 0.05; LR(15%): adjustment with stepwise logistic regression with p r = 0.15; MH: adjustment by Mantel–Haenszel stratification;PS(M): adjustment by matching on propensity score; PS(S): adjustment by stratification on propensity score; PS(R): regressionadjustment based on propensity score.72

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 25 Comparison of methods of case-mix adjustment applied to results of historically controlled studies with results of RCTsresampled from 14 regions within the ISTPercentage of studies withstatistically significantAverageVariability of results results (p < 0.05)OR SD of log OR Ratio with RCT Benefit Harm TotalRCTs 0.89 0.35 9 2 11Historically controlled studiesUnadjusted 0.88 0.44 1.23 16 4 20Stratification 0.88 0.53 1.51 17 5 22Logistic regressionFull model a 0.85 0.56 1.60 13 3 16Stepwise p r = 0.05 b 0.84 0.52 1.49 15 3 18Stepwise p r = 0.15 c 0.85 0.53 1.51 14 3 17Propensity scoreMatched d 0.91 0.43 1.23 7 2 9Stratified 0.90 0.40 1.14 9 2 11Regression 0.91 0.39 1.11 9 2 11a Full model includes 10 covariates.b Mean number of covariates included: 4.5.c Mean number of covariates included: 5.7.d Mean number of patients matched: 132 out of 200.considered by region in the last six columns ofTable 26. Stratification reduced systematic bias inonly two of the 14 regions, logistic regressionadjustments reduced bias in six regions,propensity score methods reduced bias in five ofthese six, and in two additional regions. Overall,bias introduced by use of an historical controlgroup was consistently reduced by case-mixadjustment methods in less than half of the regions.Where biases were increased by adjustment, thedirection of the increase was unpredictable. InScotland the historical control result (OR = 1.23)suggested the treatment to be harmful, in contrastto a beneficial result observed in the RCTs(OR = 0.78). Logistic regression further increasedthis bias (OR = 1.40). In Norway the oppositepattern was seen, with adjustment by logisticregression (OR = 0.41) increasing theoverestimate of treatment benefit in the historicalcontrols (OR = 0.48) compared with RCTs(OR = 0.78). However, in some regions, such asThe Netherlands, adjustment moved the historicalcontrol estimates (OR = 0.85) from a value whichwas lower than the RCTs (OR = 0.88) to a highervalue (OR = 1.14), changing a relatively correctestimate of the benefit of the intervention to abiased estimate suggestive of harm. Similar, butless extreme, changes occurred with propensityscore methods.Unpredictability in bias originating fromconcurrently controlled studiesUnpredictability in bias was observed most clearlyin the IST concurrently controlled comparisons.The ability of the seven case-mix adjustmentmethods to correct these biases is summarised inTables 27 and 28 for regional comparisons in theIST and UK city comparisons in the IST. Regionalcomparisons in the ECST are given in Table 29 forcompleteness. The results for the studiesdemonstrating the largest unpredictable biases,the regional IST comparison (see Chapter 6), arealso shown in Figure 16.As with the historically controlled studies, logisticregression increased the variability of results for allthree situations, the increased spread of resultsbeing evident in Figure 16. Use of the full logisticmodel (including all covariates) increased spreadmore than use of stepwise models. Again, therewas little corresponding increase in spurioussignificance levels as the power of the analyses wasreduced. Propensity score methods slightlyreduced unpredictable bias and spurioussignificance rates for two of the three situations,while stratification made little difference.Although there was no evidence of a systematicbias in the unadjusted results, both logisticregression and propensity score methodsintroduced small systematic biases in most73© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

74TABLE 26 Comparison of methods of case-mix adjustment applied to results of historically controlled studies with results of RCTs resampled from 14 regions within the IST, analysed by regionHistorically controlled studiesRCTs Unadjusted Logistic regression a Stratified Propensity score b% over or % over % over Average % overAverage Average or under Average or under Average or under OR or underOR OR estimate OR estimate OR estimate estimateScotland 0.78 1.23 +58 1.40 +79 1.48 +90 1.30 +67Southern Italy 0.56 1.00 +79 0.98 +75 0.89 +59 0.97 +73Northern Italy 0.94 1.15 +22 1.06 +13 1.20 +28 1.05 +12New Zealand 0.81 0.97 +20 0.94 +16 1.09 +35 1.08 +33Australia 1.30 1.39 +7 1.09 –16 1.49 +15 1.08 –17Switzerland 0.97 1.07 +10 1.05 +8 1.13 +16 1.04 +7Central Italy 0.89 0.92 +3 0.85 –4 0.89 0 0.87 –2The Netherlands 0.88 0.85 –3 1.14 +30 1.11 +26 1.10 +25Northern England 0.95 0.92 –3 0.79 –17 1.03 +8 0.86 –9Spain 1.00 0.93 –7 0.59 –41 0.57 –43 0.75 –25Southern England 1.01 0.87 –14 0.72 –29 0.85 –16 0.81 –20Poland 0.91 0.70 –23 1.08 +19 0.69 –24 1.06 +16Norway 0.78 0.48 –38 0.41 –47 0.47 –40 0.51 –35Sweden 0.94 0.44 –53 0.47 –50 0.37 –61 0.58 –38a Logistic regression was performed using the full model with 10 covariates.b Propensity scores were used in a regression adjustment.Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection bias

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 27 Comparison of methods of case-mix adjustment applied to results of concurrently controlled studies with results of RCTsresampled from 14 regions within the ISTPercentage of studies withstatistically significantAverageVariability of results results (p < 0.05)OR SD of log OR Ratio with RCT Benefit Harm TotalRCTs 0.91 0.34 7 2 9Concurrently controlled studiesUnadjusted 0.91 0.85 2.51 29 21 50Stratification 0.92 0.89 2.70 27 21 48Logistic regressionFull model a 0.92 1.12 3.39 23 18 41Stepwise p r = 0.05 b 0.91 1.01 3.06 26 20 46Stepwise p r = 0.15 c 0.91 1.03 3.12 26 20 46Propensity scoreMatched d 0.95 0.79 2.39 15 12 27Stratified 0.94 0.76 2.30 19 15 34Regression 0.95 0.75 2.27 19 15 34a Full model includes 10 covariates.b Mean number of covariates included: 4.6.c Mean number of covariates included: 5.8.d Mean number of patients matched: 101 out of 200.TABLE 28 Comparison of methods of case-mix adjustment applied to results of concurrently controlled studies with results of RCTsresampled from 10 UK cities within the ISTPercentage of studies withstatistically significantAverageVariability of results results (p < 0.05)OR SD of log OR Ratio with RCT Benefit Harm TotalRCT 1.01 0.49 7 6 13Concurrently controlled studiesUnadjusted 1.02 0.90 1.84 22 23 45Stratification 0.99 0.91 1.86 22 21 43Logistic regressionFull model a 1.05 0.96 1.96 16 17 33Stepwise p r = 0.05 b 1.03 0.89 1.82 18 20 38Stepwise p r = 0.15 c 1.04 0.89 1.82 18 20 38Propensity scoreMatched d 1.05 0.84 1.71 11 13 24Stratified 1.05 0.79 1.61 14 16 30Regression 1.03 0.74 1.51 13 15 28a Full model includes 10 covariates.b Mean number of covariates included: 3.4.c Mean number of covariates included: 4.4.d Mean number of patients matched: 109 out of 200.situations, and a large bias for concurrentlycontrolled studies from the ECST.Adjustment for bias related to ‘known’differences in case-mixIn this section we summarise the results for thethree sets of studies constructed to mimic studieswhere allocation occurred by indication.Adjusting for bias due to a known and measuredcovariateThe results of case-mix adjustment for the ISTstudies with bias relating to neurological deficit aregiven in Table 30. In these analyses systematic biaswas introduced according to values of a singlecovariate (neurological deficit). The samecovariate was then included in the models, and75© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection biasTABLE 29 Comparison of methods of case-mix adjustment applied to results of concurrently controlled studies with results of RCTsresampled from eight regions within the ECSTPercentage of studies withstatistically significantAverageVariability of results results (p < 0.05)OR SD of log OR Ratio with RCT Benefit Harm TotalRCT 1.08 0.69 3 5 9Concurrently controlled studiesUnadjusted 1.08 0.69 1.01 3 6 9Stratification 1.09 0.74 1.08 3 6 9Logistic regressionFull model a 1.20 1.43 2.10 4 6 10Stepwise p r = 0.05 b 1.06 0.82 1.21 5 7 12Stepwise p r = 0.15 c 1.07 0.91 1.34 5 8 13Propensity scoreMatched d 1.05 0.87 1.27 2 6 8Stratified 1.06 0.82 1.20 5 3 8Regression 1.06 0.80 1.17 5 3 8a Full model includes eight covariates.b Mean number of covariates included: 1.90.c Mean number of covariates included: 3.06.d Mean number of patients matched: 40/80.1005020105Odds ratio210.50.20.10.050.020.01RCTs CCs LR(F) LR(5%) LR(15%) MH PS(M) PS(S) PS(R)FIGURE 16 Comparison of methods of case-mix adjustment applied to results of concurrently controlled studies resampled from 14regions within the IST. RCT: results from corresponding randomised controlled trials; CCs: unadjusted concurrent controls withoutadjustment, LR(F): adjustment using full logistic regression analysis; LR(5%): adjustment with stepwise logistic regression withp r = 0.05; LR(15%): adjustment with stepwise logistic regression with p r = 0.15; MH: adjustment by Mantel–Haenszel stratification;PS(M): adjustment by matching on propensity score; PS(S): adjustment by stratification on propensity score; PS(R): regressionadjustment based on propensity score.76

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 30 Case-mix adjustment with bias caused by a known and measured single covariate [neurological deficit score (IST)]Percentage of studies withstatistically significantAverageVariability of results results (p < 0.05)OR SD of log OR Ratio with RCT Benefit Harm TotalRCT 0.91 0.33 7 2 9Studies where treatment is related to conditionUnadjusted 0.53 0.36 1.09 53 0 53Stratification 0.93 0.41 1.24 7 3 10Logistic regressionFull model a 0.89 0.55 1.67 8 3 11Stepwise p r = 0.05 b 0.87 0.51 1.55 12 3 15Stepwise p r = 0.15 c 0.88 0.52 1.58 10 3 13Propensity scoreMatched d 0.93 0.45 1.36 4 2 6Stratified 0.90 0.40 1.21 6 2 8Regression 0.92 0.39 1.18 5 2 7a Full model includes 10 covariates (including neurological deficit score).b Mean number of covariates included: 4.6.c Mean number of covariates included: 5.9.d Mean number of patients matched: 127 out of 200.TABLE 31 Case-mix adjustment with bias caused by a known but unmeasured single covariate [consciousness level (IST)]Percentage of studies withstatistically significantAverageVariability of results results (p < 0.05)OR SD of log OR Ratio with RCT Benefit Harm TotalRCT 0.91 0.33 7 2 9Studies where treatment is related to conditionUnadjusted 0.64 0.34 1.03 34 0 34Stratification 0.70 0.38 1.15 23 0 23Logistic regressionFull model a 0.71 0.48 1.45 17 1 18Stepwise p r = 0.05 b 0.69 0.43 1.30 21 1 22Stepwise p r = 0.15 c 0.69 0.44 1.33 20 1 21Propensity scoreMatched d 0.79 0.38 1.15 9 0 9Stratified 0.77 0.34 1.03 12 0 12Regression 0.78 0.34 1.03 11 0 11a Full model includes 10 covariates (excludes consciousness).b Mean number of covariates included: 4.0.c Mean number of covariates included: 5.3.d Mean number of patients matched: 134 out of 200.used as the variable for stratification. The bias wasappropriately adjusted for by all seven methods.Adjustments made by stratification gave resultsclosest to those of the RCTs.Adjusting for bias due to a known butunmeasured covariateTable 31 reports results of case-mix adjustment forthe bias in the IST trial introduced whentreatment allocation was linked to the level ofconsciousness at admission. However, in thissituation, the adjustments were made without thecovariate indicating level of consciousness beingincluded in the models. As before, stratificationwas undertaken using the neurological deficitscore, but the regression and propensity scoremodels were based on the remaining nine out of10 covariates.77© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection biasTABLE 32 Case-mix adjustment with bias caused by multiple covariates, some measured some unmeasured, with unknownmechanism (based on observed outcomes in the IST)Percentage of studies withstatistically significantAverageVariability of results results (p < 0.05)OR SD of log OR Ratio with RCT Benefit Harm TotalRCT 0.91 0.33 7 2 9Studies where treatment is related to conditionUnadjusted 0.51 0.34 1.03 60 0 60Stratification 0.51 0.38 1.15 54 0 54Logistic regressionFull model a 0.45 0.50 1.52 50 0 50Stepwise p r = 0.05 b 0.47 0.44 1.33 51 0 51Stepwise p r =0.15 c 0.47 0.46 1.39 51 0 51Propensity scoreMatched d 0.58 0.37 1.12 31 0 31Stratified 0.57 0.34 1.03 39 0 39Regression 0.57 0.33 1.00 39 0 39a Full model includes 10 covariates.b Mean number of covariates included: 4.5.c Mean number of covariates included: 5.8.d Mean number of patients matched: 137 out of 200.78All methods adjusted the crude estimate of thetreatment effect (OR = 0.64) in the direction ofthe result of the RCTs (OR = 0.91), thus removingsome of the selection bias. However, stratificationand logistic regression (LR) removed only a smallfraction of the bias (OR for LR full model = 0.71,OR for stratification = 0.70), propensity score (PS)methods did somewhat better (OR for a matchedPS of 0.79), but remained substantially biased andhence gave far too many statistically significantresults. While the selection mechanism was notdesigned to introduce an unpredictable bias, theresults from the logistic regression model weremuch more variable than the unadjustedresults.Adjusting for bias due to unknown multiplecovariatesSelection according to outcome, as anticipated,introduced strong biases into the data. For theIST, the non-randomised unadjusted results(OR = 0.51) significantly overestimated treatmentefficacy compared with the RCT results(OR = 0.91) (Table 32).Stratification failed to adjust for the bias at all,with the results being identical with theunadjusted results. Significance rates decreasedslightly.Adjustment using logistic regression increasedbias. For the IST the unadjusted average OR of0.51 decreased to 0.45 (full model). Variability ofresults also increased, the distribution of adjustedresults being 1.48 times the variability ofunadjusted results for the IST. Significance ratesdecreased slightly.PS methods slightly reduced bias in the IST. Thevariability of results increased, but not as much asfor logistic regression results, whilst significancerates decreased.Discussion“The first experience with multivariate analysis is aptto leave the impression that a miracle in thetechnology of data analysis has been revealed; themethod permits control for confounding andevaluation of interactions for a host of variables withgreat statistical efficiency. Even better, a computerdoes all the arithmetic and neatly prints out theresults. The heady experience of commanding acomputer to accomplish all these analytic goals andthe simply gathering and publishing the sophisticated‘output’ with barely a pause for retyping is undeniablyalluring. However useful it may be, multivariateanalysis is not a panacea. The extent to which thisprocess represents improved efficiency rather thanjust bias depends on the adequacy of the assumptionsbuilt into the mathematical model.”From Rothman 149The results of our investigations can besummarised by the following four key results, all of

Health Technology Assessment 2003; Vol. 7: No. 27which raise concerns about the performance ofcase-mix adjustment methods:1. Comparisons between non-randomised groupsthat appear comparable in terms of case-mixare often biased, sometimes more than for nonrandomisedgroups that do not appearcomparable.2. Case-mix adjustment methods rarelyadequately adjust for differences in case-mix.3. Logistic regression always increases variabilityin study results.4. All adjustment methods can on occasionincrease systematic bias.The first and second observations are notsurprising, and have been discussed before. 150 Thethird observation has been demonstratedtheoretically always to be the case, 151 although wesuspect that this result is not well known. Thefourth observation is contrary to most beliefsabout case-mix adjustment methods, and has onlybecome detectable through the unique resamplingdesign of our investigation.As Rothman described above, statistical riskadjustment methods are alluring, appearing toprovide a simple solution to many of theinadequacies of the design and execution of nonrandomisedstudies. This message has been widelydisseminated throughout the medical researchcommunity, leading to their routine use inepidemiology and health services research. 136However, the validity of a risk-adjustment modeldepends on fulfilling a demanding set ofassumptions. Below we consider the assumptionsthat may be most critical.Why adjustment methods might notwork?Omitted covariatesRisk adjustment models can only adjust fordifferences in variables that have been observedand measured. As seen with the ‘allocation byindication’ mechanisms based on neurologicaldeficits (IST) (Table 30), if all variables linked tothe allocation mechanism have been measuredand observed, then adequate adjustment may bemade. However, in most situations we do not knowthe variables upon which allocation is based.There may be important prognostic factors thatthe investigators do not know about or have notmeasured which are unbalanced between groupsand responsible for differences in outcome. Forexample, when allocation may be influenced bylevel of consciousness (IST) (Table 31) but theconsciousness variable was not included in themodel, inadequate adjustment for bias was made.If the missing covariates affecting allocation arecorrelated with the observed covariates, somedegree of adjustment is likely to be observed (aswas seen for degree of consciousness), but it isunlikely to be adequate unless the correlations arevery strong. Many texts refer to the unadjustedeffect as ‘residual confounding’ or ‘hidden bias’.Rosenbaum proposed a strategy for investigatingthe robustness of an observational finding ofhidden bias based on sensitivity analyses whichdetermine the size of covariate associationrequired to nullify an observed treatment effect. 152In some situations this approach could help todecide whether hidden bias could fully explain anobserved effect, but such an assessment retains adegree of subjective judgement.Our first and second results could be explained bythe common situation of treatment assignmentdepending on unmeasured covariates. However,missing covariates cannot explain the third andfourth results.Misspecified continuous covariates and omittedinteractionsTwo forms of misspecification can occur in casemixadjustment analyses. The first is when acontinuous variable is categorised, or a categoricalvariable is regrouped into a smaller number ofcategories. Cochran 153 presented analyticalinvestigations that showed that for a continuouscovariate related to outcome with a monotonictrend, dichotomisation would leave 36% of thevariability caused by the relationship unexplained(subject to some distributional assumptions).His results suggest that five categories areneeded to explain successfully at least 90% of thevariability. Brenner and Blettner 154 extendedthis work to consider the efficiencies of differentapproaches of modelling various monotonictrends using multiple categorisations and linearterms.However, Brenner 155 also showed that if thecovariate does not have a monotonic effect, correctcategorisation of the data can be crucial toobtaining a sensible result. Table 33 presentshypothetical data taken from Brenner’s paperwhere there is a U-shaped relationship betweenthe covariate and the outcome. Adjusting for thecovariate classified in three categories (Table 33a)makes an appropriate adjustment, the observedtreatment effect being OR = 1 in all categories.However, if the risk factor is dichotomiseddifferent results are obtained depending on wherethe dichotomisation is made: in Table 33(b) the79© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

80TABLE 33 Hypothetical example demonstrating the potential impact of crudely classifying a covariate(a) Variable classified in three categoriesCV = 1Dead Alive TotalT 20 10 30C 160 80 240Total 180 90 270OR = 1(b) Variable dichotomised at CV = 1CV = 1Dead Alive TotalT 20 10 30C 160 80 240Total 180 90 270OR = 1(c) Variable dichotomised at CV = 2CV < 2Dead Alive TotalT 40 120 160C 180 190 370Total 220 310 530OR = 0.35CV = 2Dead Alive TotalT 20 110 130C 20 110 130Total 40 220 260OR = 1CV > 1Dead Alive TotalT 36 118 154C 22 111 133Total 58 229 287OR = 1.54CV = 3Dead Alive TotalT 16 8 24C 2 1 3Total 18 9 27OR = 1CV = 3Dead Alive TotalT 16 8 24C 2 1 3Total 18 9 27OR = 1Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection biasAdapted from Brenner, 1997 155 . C, control group; CV, covariate; T, treatment group.

Health Technology Assessment 2003; Vol. 7: No. 27stratified estimate of the OR is >1 and inTable 33(c) the stratified estimate of the OR is >1.Misspecification of relationships for continuous orordinal covariates could therefore also (partly)explain the lack of adjustment we observed (Result2). It is also possible that misspecification of aU-shaped relationship as a dichotomy or a lineartrend could explain the increase in bias observedwith adjustment (Result 4). However, it seemsunlikely that this is the full explanation of theseresults as there are few true continuous or ordinalcovariates in the analyses, and U-shapedrelationships are rare.The second type of misspecification is theomission of interactions between covariates, orbetween covariates and allocation. If covariates dointeract, omitting the interaction term from acase-mix adjustment model will lead to areduction in the potential adjustment that can bemade, hence increasing residual confounding isanother possible explanation of our second result.Interactions between covariates and the treatmentallocation are more complicated, and imply thatthe treatment effect should not be summarised asa single value but as a set of values dependent onthe covariate. We have not considered this level ofcomplexity for either the results of RCTs or nonrandomisedstudies (despite this being likely forthe ECST, where degree of stenosis is known torelate to benefit of carotid endarterectomy). Whenthese interactions are omitted, the effect oftreatment is summarised as a single ‘average’value, the average depending on the distributionof covariates of participants included in the study.As the omission of interactions applied to both theRCTs and the non-randomised studies, it isunlikely to be the key explanation of the resultsthat we observed.Multicollinearity among covariatesIf a model includes two or more covariates whichare strongly correlated with each other, it isdifficult to disentangle their effects. In thissituation, it has often been noted that theestimated regression coefficients may changedrastically according to what other variables areincluded in the model, which is disconcerting. Inaddition, the estimates of effect are likely to bemade with low precision. Such multicollinearityamong covariates may not be too much of aproblem in a non-randomised study unless theycause the adjustment procedures to ‘explode’,producing infeasibly small or large estimates ofparameters or standard errors. However,multicollinearity between a covariate and thetreatment allocation will make it impossible toseparate the independent effects of the covariateand of treatment on outcome. 156 The allocationrelationships in non-randomised studies are rarelystrong enough for this to be a major concern, andthis is unlikely to be an explanation for the resultsthat we observed.Misclassification and measurement errorWe have already noted that in order to be able toadjust for a confounding factor, it must bemeasured and included in the adjustment model.To do this it is important (a) that the realconfounding factor is used, and not a surrogate orproxy variable, and (b) that it is measured withouterror. Covariate misclassification through the useof poor proxies, measurement error and withinparticipant instability in covariates (e.g. because ofcircadian rhythms) all lead to underestimation ofthe effect of each covariate on outcome. Thisphenomenon, often known as regression dilutionor attenuation, 157,158 is demonstrated usinghypothetical data in Table 34. The first rowpresents data from a non-randomised studystratified according to the true values of twocovariates, where the underlying treatment effectis OR = 0.5. Owing to the confounding effect ofthe first covariate, the observed result is biased:OR = 1.28 (Table 34a). Adjusting for the firstcovariate corrects for this bias (OR = 0.50).However, if the first covariate is assessed withmeasurement error, we could observe thecategorisation observed in Table 34(b). Here,adjustment for the same covariate has much lesseffect, the adjusted estimate being OR = 1.17.The impact of misclassification in this example islarge, but is generated by a typical degree ofmisclassification corresponding to a kappacoefficient of 0.4, which is routinely interpreted asshowing ‘fair to moderate’ agreement.Methods have been developed to correct forregression dilution and are widely used throughoutepidemiology. 158–160 However, they require thedegree of misclassification to be estimated, eitherwithin the study or in additional reliabilityinvestigations, and their ability to correct formisclassification depends on the precision withwhich these estimates are made. When there aremultiple sources of misclassification, correction formeasurement error becomes complex. 160,161Where confounders cannot be corrected formisclassification, it is often assumed thatadjustment using the covariate will adjust only81© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

82TABLE 34 Hypothetical example demonstrating the potential impact of adjusting for covariates when misclassifications are correlatedObserved unstratified resultsDead Alive TotalT 165 201 366C 143 223 366Total 308 424 732Unadjusted OR = 1.28Dead Alive TotalT 165 201 366C 143 223 366Total 308 424 732Unadjusted OR = 1.28Dead Alive TotalT 165 201 366C 143 223 366Total 308 424 732(a) Results stratified according to underlying distribution of covariates (unobserved) aCV1 = 0, CV2 = 0Dead Alive TotalT 6 24 30C 100 200 300Total 106 224 330OR = 0.5CV1 = 1, CV2 = 0Dead Alive TotalT 30 30 60C 40 20 60Total 70 50 120OR = 0.5True underlying OR = 0.50, OR adjusted only for CV1 (observed without misclassification) = 0.50(b) Results stratified according to covariates with misclassification b in CV1 aCV1 = 0, CV2 = 0Dead Alive TotalT 13 26 39C 82 146 228Total 95 172 267OR = 0.89CV1 = 1, CV2 = 0Dead Alive TotalT 23 28 51C 58 74 132Total 81 102 183OR = 1.05CV1 = 0, CV2 = 1Dead Alive TotalT 6 24 30C 1 2 3Total 7 26 33OR = 0.5CV1 = 0, CV2 = 1Dead Alive TotalT 41 54 95C 1 2 3Total 42 56 98OR = 1.52OR adjusted only for CV1 (observed with misclassification) = 1.17CV1 = 1, CV2 = 1(c) Results stratified according to covariates with additional misclassification c in CV2 for those misclassified on CV1 aCV1 = 0, CV2 = 0Dead Alive TotalT 50 63 113C 83 146 229Total 133 209 342CV1 = 1, CV2 = 0Dead Alive TotalT 21 21 42C 28 14 42Total 49 35 84CV1 = 0, CV2 = 1Dead Alive TotalT 4 17 21C 1 1 2Total 5 18 23Dead Alive TotalT 123 123 246C 2 1 3Total 125 124 249OR = 0.5CV1 = 1, CV2 = 1Dead Alive TotalT 88 93 181C 2 1 3Total 90 94 184OR = 0.47CV1 = 1, CV2 = 1Dead Alive TotalT 90 100 190C 31 62 93Total 121 162 283Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection biasUnadjusted OR = 1.28OR = 1.40OR = 0.5OR = 0.24OR = 1.80OR additionally adjusted for CV2 (observed with correlated misclassification) = 1.32a Adjusted OR obtained from Mantel–Haenszel estimators.b Misclassification in CV1: 30% of CV1=0 misclassified as CV1=1, 30% of CV1=1 misclassified as CV1=0.c Misclassification in CV2: CV2 always classified as CV2=1 if CV1 misclassified as CV1=1; CV2 always classified as CV2=0 if CV1 misclassified as CV1=0.CV, covariate; CV1, first covariate; CV2, second covariate.

Health Technology Assessment 2003; Vol. 7: No. 27partially for the effect of the covariate. The logicalcorollary of this assumption is that it is alwaysworthwhile to adjust for a covariate, howeverpoorly it has been measured. However, this notionin turn depends critically on the assumption thatmisclassification in the covariate is itselfunbiased. 157 Greenland and Robins 162 discussedscenarios for case-control studies where thisassumption is relaxed. They considereddifferential misclassification in disease, exposureand a single covariate, and showed that in somemisclassification scenarios it can actually bedetrimental to adjust for the covariate – theunadjusted estimate may be closer to theunderlying effect than the adjusted estimate. Inthese situations it is not a matter of adjustmentbeing inefficient but rather that adjustment leadsto totally incorrect conclusions. Such an error willbe magnified by the additional credence oftengiven to an adjusted analysis when reporting studyresults.Greenland and Robins’ scenarios hint at amechanism that might explain our troublingfourth key result. However, their scenarios do nottranslate directly to consideration of nonrandomisedintervention studies: they consideredmeasurement error in the exposure (treatment)and case–control status (outcome). They also onlyassessed the impact of measurement error in asingle covariate, whereas case-mix adjustmentinvolves many covariates, all of which could besubject to misclassification. We have extended thehypothetical example in Table 34 to includestratification for a second covariate (CV2). InTable 34(c) we have introduced misclassification inCV2 but only for those misclassified for CV1. Weobserve that the OR adjusted for observed CV1and CV2 (OR = 1.32) is now further from theunderlying true value (OR = 0.5) than theunadjusted estimate (OR = 1.28). Hence such amechanism could explain our fourth observation,that adjustment in some situations can increasebias in estimates of treatment effects.If misclassification was solely a matter ofmeasurement error, such correlated misclassificationswould be rare. However, misclassification includesall processes that lead us to observe a value thatdiffers from the true value of the confounder. Forexample, consider a study in which the trueconfounder is a person’s average blood pressure.The blood pressure reading used in the analysiscould be misclassified owing to measurementerror, but it could also be misclassified owing tonatural within-person variation in blood pressurerelated to the time of day when the measurementwas made, or to a short period of stress. If asecond variable was included in the analysis thatwas also influenced by circadian rhythm or stress,misclassification in the second variable will becorrelated with misclassification in blood pressure.If case-mix adjustment models are beinginfluenced by correlated misclassification, it couldbe hypothesised that including more variables inan adjustment model will increase the chances ofcorrelated misclassifications, and hence increaseexpected bias. Comparison of the three differentlogistic models across Tables 24, 25, 27–29 and32 potentially supports this hypothesis: in nearlyall cases the logistic regression model includingthe most covariates (the ‘full model’) was morebiased than the logistic regression model with thefewest covariates (backward stepwise withp-to-remove of 0.05).Misclassification and measurement error incovariates has been cited as the possible root ofmany controversial associations detected throughobservational studies. 163 We are correspondinglyconcerned that measurement and misclassificationerrors may be largely responsible for our fourthresult.Differences between unconditional andconditional estimatesCovariates which are prognostic but balancedacross groups are rarely routinely adjusted for inanalyses of randomised controlled trials. Evenwhen adjustments are made, the adjustedestimates are often ignored when results of trialsare included in meta-analyses. We have followedthis standard approach in calculating the resultsof the RCTs that we created from the IST andECST data sets. These unadjusted estimates areknown as unconditional or population averageresults.Estimates of treatment effects where prognosticvariables have been adjusted for are known asconditional estimates, being conditional onknowledge of the covariates included in the analysis.Our comparison of results of RCTs with adjustedresults of non-randomised studies is thereforecomparing unconditional estimates of treatmenteffects from RCTs with conditional estimates oftreatment effects from non-randomised studies.When using ORs, it has been noted thatadjustment for prognostic covariates leads todifferences between unconditional and conditionalestimates of treatment effects, even for covariatesthat are balanced across treatment groups. 164,16583© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection biasTABLE 35 Hypothetical example demonstrating the potential impact of not adjusting for a balanced prognostic covariate in an RCTCrude analysisDead Alive TotalT 140 60 200C 60 140 200Unconditional estimate of OR = 5.4Adjusted analysisCV = 0Dead Alive TotalT 90 10 100C 50 50 100Stratum-specific estimate of OR = 9CV = 1Dead Alive TotalT 50 50 100C 10 90 100Stratum-specific estimate of OR = 9Conditional estimate of OR = 9Adapted from Gail, 1984. 16484Consider the trial in Table 35. The unconditionalestimate of the treatment effect is OR = 5.4. Thelower half of the table shows the results of thesame trial stratified by a prognostic covariate thatis perfectly balanced across treatment groups. Theestimate of the treatment effect in each strata isOR = 9. Thus the estimate of the treatment effectconditional on knowledge of the covariate isOR = 9. It can be deduced that if there were afurther balanced prognostic covariate to adjust for,the result would change further, always movingfurther from the null effect value of OR = 1. 164This conditional result would be obtained throughadjustment using both logistic regression andstratification. However, the propensity score forparticipants in the hypothetical trial is 0.5regardless of their covariate value, and thereforethe estimates of the treatment effect usingpropensity score methods will be OR = 5.4 – theunconditional estimate. Propensity scores methodsonly make adjustments for covariates that are notbalanced across treatment groups.Hence the difference between unconditional andconditional results is one possible explanation ofthe differences observed between RCT results andthe results of the logistic regression adjustedanalyses of non-randomised studies, and alsobetween the results of adjustment using logisticregression and adjustment using propensity scoremethods.Comparison of methodsStratificationStratification is best used to adjust for a singlecovariate. When stratification is used for severalcovariates, the strata become numerous and sosmall in size that many of the cells contain onlytreated participants or control participants, orparticipants all of whom have the same outcomestate. In these situations the strata do notcontribute to the analysis, and the data from thoseparticipants are effectively discarded. Even so,when bias relates to a single ordinal covariate,stratification can yield the best adjustment (as wasseen in Table 30) as stratification estimates aseparate parameter for each category, avoidingspecifying a trend across categories to be eitherlinear or monotonic. As selection bias rarelyrelates to a single variable, stratification will eitherbe an inefficient (if multiple covariates arestratified) or an inadequate (if only one covariateis stratified) method for adjusting for differencesin case-mix in non-randomised studies.Logistic regressionIn clinical trials, covariate adjustment is oftenrecommended as a method of improving theprecision of an estimate of treatment effect, even ifthere is no overt imbalance between the groups.This result, however, is particular to the use oflinear regression and continuous outcomemeasures. Robinson and Jewell have shown thatlogistic regression always leads to a loss of

Health Technology Assessment 2003; Vol. 7: No. 27precision. 151 Their theoretical finding explains theincreased variability of adjusted results that weobserved with all applications of logisticregression, which we have interpreted as increasedunpredictability in bias. However, unlike theincreased variability observed with historically andconcurrently controlled non-randomised studies inChapter 6, the standard errors of the adjustedestimates are also inflated, such that the extraincreased variability does not further increasespurious statistical significance rates.One dilemma in all regression models is the processby which covariates are selected for adjustment.Many texts discuss the importance of combiningclinical judgement and empirical methods toensure that the models select and code variables inways that have clinical face validity. There arethree strategies that are commonly used in healthcare research to achieve this, described below.Recently there has been a trend to include allscientifically relevant variables in the model,irrespective of their contribution to the model. 166The rationale for this approach is to provide ascomplete control of confounding as possiblewithin the given data set. This idea is based on thefact that it is possible for individual variables notto exhibit strong confounding, but when takencollectively considerable confounding can bepresent in the data. One major problem with thisapproach is that the model may be overfitted andproduce numerically unstable estimates. However,as we have observed, a more important problemmay be the increased risk of including covariateswith correlated misclassification errors.The stepwise approaches to selecting covariatesare often criticised for using statistical significanceto assess the adequacy of a model rather thanjudging the need to control for specific factors onthe basis of the extent of confounding involved,and in using sequential statistical testing, known tolead to bias. 167 Research based on simulations hasfound that stepwise selection strategies which usehigher p-values (0.15–0.20) are more likely tocorrectly select confounding factors than thosewhich use a p-value of 0.05. 168,169 In ourevaluations, little practical difference was observedbetween these two stepwise strategies.A pragmatic strategy for deciding which estimatesto adjust for involves undertaking unadjusted andadjusted analyses and using the results of theadjusted analysis when they differ from those ofthe unadjusted analysis. This is based on anargument that if the adjustment for a covariatedoes not alter the treatment effect the covariate isunlikely to be important. 141 An extension of thisargument is used to determine when all necessaryconfounders have been included in the model,suggesting that confounders should keep beingadded to a model so long as the adjusted effectkeeps changing (e.g. by at least 10%). Theassumed rationale for this strategy sometimesmisleads analysts to reach the unjustifiedconclusion that when estimates become stable allimportant confounders have been adjusted for,such that the adjusted estimate of the treatmenteffect is unbiased. We did not attempt to automatethis variable selection approach in our evaluations.Propensity score methodsPropensity score methods are not widely used inhealthcare research, and are difficult to undertakeowing to the lack of suitable software routines.However, there may be benefits of the propensityscore approach over traditional approaches inmaking adjustments in non-randomised studies.Whilst Rosenbaum and Rubin showed that for biasintroduced through a single covariate thepropensity score approach is equivalent to directadjustment through the covariate, 146 our analyseshave shown that when there are multiplecovariates the propensity score method may in factbe superior as it does not increase variability inthe estimates. In addition, propensity scoremethods give unconditional (or populationaverage) estimates of treatment effects, which aremore comparable to typical analyses of RCTs.Simulation studies have also shown that propensityscores are less biased than direct adjustmentmethods when the relationship of covariates ismisspecified. 170The impact of misclassification and measurementerror on propensity score methods appears not tohave been studied. It is unclear whether theseproblems can explain the occasionalovercorrection of propensity score methods thatwe observed. Also, our implementation of thepropensity score method did not includeinteraction terms in the estimation of propensityscores, as is sometimes recommended. 147 It wouldbe interesting to evaluate whether includingadditional terms would have improved theperformance of the model.ConclusionsThe problems of underadjustment forconfounding are well recognised. However, in anon-randomised study it is not possible to assess85© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Empirical evaluation of the ability of case-mix adjustment methodologies to control for selection biasdirectly the likely degree of residual confoundingthat may be present, and therefore we cannotgauge how biased adjusted results may still be. Bycomparing with results based on randomisation,our investigations suggest that the degree ofunderadjustment may be large. Indeed, ourresults may in fact be overoptimistic, as thecovariate data used were recorded in a standardway according to trial protocols, and werecomplete for all participants. In many nonrandomisedstudies measurement methods arenot standardised. Also, covariate data areincomplete (especially in retrospective studies),leading to bias if the observations are not missingat random.Our two greatest concerns are the potentialincrease in bias that could occur as a result of theexistence of correlated misclassification ofcovariates, and the differences betweenconditional and unconditional estimates.Correlated misclassification is a problem inherentto the data, and cannot be adjusted for. It is verydifficult to know the degree of misclassificationand error in a variable, and impossible to knowwhether the variable being used is the ‘true’confounder or just a proxy. These findingsquestion the appropriateness of the strategy ofincluding data on all available potentialconfounders when adjusting for case-mix, whichhas been the starting point of many riskadjustmentmethods used throughout healthcare.However, the same findings could be explained bythe peculiar differences between unconditionaland conditional estimates of treatment effectsobserved when results are expressed as ORs,although this mechanism only applied to estimatesobtained from logistic regression and stratificationmethods.The finding of high levels of residual confoundingand the detrimental effect of adjustment were seenin both historically controlled studies, known to beprone to systematic bias, and in concurrentlycontrolled studies, more prone to unpredictabilityin bias. The relationships were also noted instudies mimicking allocation by indication.It is important to find out whether suchdestructive relationships between covariates arecommon. We have examined data from only twoclinical situations, but in both we observed resultsthat undermine the use of case-mix adjustment.Also in the IST, case-mix adjustment was found tobe detrimental in eight of the 14 regions.There appears to be a small potential benefit ofusing propensity score methods over logisticregression for case-mix adjustment in terms of theconsistency of estimates of treatment effects. Whilelogistic regression always increased the range ofobserved treatment effects, propensity scoremethods did not. This finding may indicate agreater role for propensity score methods inhealthcare research, although in the particularapplications investigated neither approachperformed adequately.For those critically appraising non-randomisedstudies, the recommendation to assess whether“investigators demonstrate similarity in all knowndeterminants of outcome” 138,139 has not beenuniversally supported by our empiricalinvestigations. The second recommendation, toassess whether they “adjust for these differencesin analysis” is also not supported empirically.Our analyses suggest that there are considerablecomplexities in assessing whether a casemixadjustment analysis will increase ordecrease bias.These findings may have a major impact on thecertainty which we assign to many effects inhealthcare which have been made on the basis ofusing risk adjustment methods.86

Health Technology Assessment 2003; Vol. 7: No. 27Chapter 8Discussion and conclusionsChapters 3–7 have reported results from fiveseparate evaluations concerning non-randomisedstudies. The results have been discussed in detailin each chapter. We summarise their main findingsbelow.Summary of key findingsOur review of previous empirical investigations ofthe importance of randomisation (Chapter 3)identified eight studies that fulfilled our inclusioncriteria. Each investigation reported multiplecomparisons of results of randomised and nonrandomisedstudies. Although there was overlap inthe comparisons included in these reviews, theyreached different conclusions concerning the likelyvalidity of non-randomised data, mainly reflectingweaknesses in the meta-epidemiologicalmethodology that they all used, most notably thatit was not able to account for confounding factorsin the comparisons between randomised and nonrandomisedstudies, nor to detect anything otherthan systematic bias.We identified 194 tools that could be used toassess the quality of non-randomised studies(Chapter 4). Overall the tools were poorlydeveloped: the majority did not provide a meansof assessing the internal validity of nonrandomisedstudies and almost no attention waspaid to the principles of scale development andevaluation. However, 14 tools were identified thatincluded items related to each of our pre-specifiedcore internal validity criteria, which related toassessment of allocation method, attempts toachieve comparability by design, identification ofimportant prognostic factors and adjustment ofdifferences in case-mix. Six of the 14 tools wereconsidered potentially suitable for use as qualityassessment tools in systematic reviews, but allrequire some modification to meet all of our prespecifiedcriteria.Of 511 systematic reviews we identified thatincluded non-randomised studies, only 169 (33%)assessed study quality, and only 46% of thesereported the results of the quality assessment foreach study (Chapter 5). This is lower than the rateof quality assessment in systematic reviews ofrandomised controlled trials. 131 Among those thatdid assess study quality, a wide variety of qualityassessment tools were used, some of which weredesigned only for use in evaluating RCTs, andmany were designed by the review authorsthemselves. Most reviews (88%) did not assess keyquality criteria of particular importance for theassessment of non-randomised studies. Sixty-ninereviews (41%) investigated the impact of quality onstudy results in a quantitative manner. The resultsof these analyses showed no consistent pattern inthe way that study quality relates to treatmenteffects, and were confounded by the inclusion of avariety of study designs and studies of variablequality.A unique ‘resampling’ method was used togenerate multiple unconfounded comparisonsbetween RCTs and historically controlled andconcurrently controlled studies (Chapter 6). Theseempirical investigations identified twocharacteristics of the bias introduced by using nonrandomallocation. First, the use of historicalcontrols can lead to systematic over- orunderestimations of treatment effects, thedirection of the bias depending on time trends inthe case-mix of participants recruited to the study.In the studies used for the analyses, these timetrends varied between study regions, and weretherefore difficult to predict. Second, the results ofboth study designs varied beyond what wasexpected from chance. In a very large sample ofstudies the biases causing the increasedunpredictability on average cancelled each otherout, but in individual studies the bias could befairly large, and could act in either direction.These biases again relate to differences in casemix,but the differences are neither systematic norpredictable.Four commonly used methods of dealing withvariations in case-mix were identified: (i) discardingcomparisons between groups which differ in theirbaseline characteristics, (ii) regression modelling,(iii) propensity score methods and (iv) stratifiedanalyses (Chapter 7). The methods were applied tothe historically and concurrently controlled studiesgenerated in Chapter 6, and also to studiesdesigned to mimic ‘allocation by indication’. Noneof the methods successfully removed bias in87© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Discussion and conclusions88historical and concurrent cohort studies. Logisticregression in fact tended to increase the bias.Propensity score methods performed slightly betterthan other methods, but did not yield satisfactoryadjustments in most situations. Detailedinvestigation revealed that adequate adjustment forselection bias could only be made when selectiondepends on a single prognostic factor that ismeasured and included in the adjustment model.Although apparent under-adjustment could beexplained by omission of important confoundingfactors, the observation that adjustment could alsoincrease bias required different explanations. Ofpossible explanations identified, we considered themost likely to be the difference betweenunconditional and conditional estimates of ORsand the inclusion of confounders in the adjustmentmodels that have correlated mismeasurement ormisclassification errors, and the differences betweenconditional and unconditional estimates.Discussion“Scientific evidence is commonly and properlygreeted with objections, scepticism and doubt.Responsible scientists are responsibly sceptical. Welook for failures of observation, gaps in reasoning andalternative explanations. This scepticism is itselfscrutinised. Scepticism must itself be justified,defended. One needs ‘grounds for doubt’.”From Rosenbaum 145Non-randomised studies are widely usedthroughout healthcare to evaluate the intendedeffects of healthcare interventions. They have beenincluded in many systematic reviews and, onoccasion, used as the sole basis for healthcaredecisions and policy. While they are widelyperceived to have greater ‘external validity’ thanRCTs, 25 their internal validity is questionable,principally owing to problems of selection bias,although they often have weaknesses in other areas.These weaknesses lead many to be sceptical aboutthe validity of their results. In this project we haveattempted to evaluate the degree to which thisscepticism is justified, through reviews of existingevidence and through new empirical investigations.Two studies recently published in the New EnglandJournal of Medicine (NEJM) 32,33 challenged theexisting evidence base that has accumulated aboutthe validity of non-randomised studies forassessing the intended effects of healthcareinterventions. These two studies and, to a largeextent, the research of the five other groupsreviewed in Chapter 3, attempted to answer thesimple question, ‘are non-randomised studiesbiased?’. Between them, these reviews haveaccumulated many instances where the results ofrandomised and non-randomised studies of thesame intervention are on average the same, butalso many examples where they differ. Althoughnot the conclusion of all the authors, the oneconclusion that fits with all previous research andis supported by our new empirical investigations(Chapter 6) is that non-randomised studies aresometimes but not always biased. Havingestablished that bias is a likely possibility, it is ofmore importance to understand (a) what causesbias in non-randomised studies, (b) how often andhow badly biased are the results from nonrandomisedstudies, (c) whether the presence ofbias can in any way be predicted and (d) what theimplications are for those producing, using andreviewing non-randomised studies.What are the causes of bias in nonrandomisedstudies?The principal difference between randomised andnon-randomised studies can be characterised as adifference in their susceptibility to selection bias,that is, a bias which acts such that participantsselected to receive one intervention are in someway different from those selected to receive thealternative intervention. Concealed randomisationspecifically removes the possibility of selectionbias, the only differences between the outcome ofdifferent groups being attributable to chance or tothe intervention, all else being equal. In nonrandomisedstudies, allocation to groups dependson other factors, sometimes known, sometimesunknown. When these factors are also related tooutcome, bias will be introduced.There are other issues that commonly lead to biasin non-randomised studies. For example, numbersof exclusions in non-randomised studies arefrequently unclear, treatment and outcomeassessment are rarely conducted according tostandardised protocols and outcomes may not beassessed blind. None of these issues areinsurmountable in a prospective non-randomisedstudy, but while the threat of increased selectionbias can be reduced, it can never be removed.How often and how biased are nonrandomisedstudies?Our resampling studies have demonstrated thatthe biases associated with historically andconcurrently controlled non-randomised studiesare large enough to impact on the conclusions of asystematic review. Some 50% of the concurrentlycontrolled non-randomised studies sampled fromthe IST met standard criteria for statistical

Health Technology Assessment 2003; Vol. 7: No. 27significance compared with only 9% of comparableRCTs. Taking a significant result as leading to aconclusion of effectiveness, selection bias in theseconcurrently controlled studies was strong enoughfor 41% to reach unjustified conclusions. Notably,21% of these studies wrongly concluded thataspirin was harmful. Similarly, 40% of the ECSThistorically controlled studies were statisticallysignificant compared with 15% of RCTs, implyingthat the use of historical comparisons introducedbias severe enough to lead to unjustifiedconclusions in favour of surgery in 25% of thestudies. This high frequency of ‘conclusionchanging’bias raises concern.However, are these frequencies and magnitudes ofbias likely to be typical? In many clinical situationsthe variation in case-mix is large, as, for example,in the treatment of those who have suffered astroke, who, as seen in the IST trial, vary in theirlevels of consciousness, paralysis and many otherprognostic factors. Here the possibility ofproducing groups with large differences in casemix(resulting in serious bias) is high, such thatthe results of non-randomised comparisons areunlikely to be reliable. In other situations there islittle prognostic information available at the timeof treatment allocation, so that the possibility ofintroducing case-mix related biases might belower. For example, in childhood vaccination trialsin the developed world there is little possibility ofpredicting the occurrence of infectious disease atthe time of vaccine administration. It couldtherefore be hypothesised that non-randomisedstudies in such contexts would be less likely to bebiased. These issues require further research.Can we predict bias?One issue that is clearly important when trying topredict bias is the quality of the study design.Whilst there is hardly any empirical evidenceabout the importance of particular dimensions ofstudy quality for non-randomised studies, theimportance of evaluating the quality of RCTs forthe likelihood of performance, attrition anddetection bias has been well established, and thereis good empirical evidence of bias associated withcertain methodological features. 23 Despite the lackof empirical evidence specific to non-randomisedstudies, it is likely that they are equally if not moresusceptible to these biases than RCTs.It has been suggested that to assess the likelihoodof selection bias in a non-randomised study, it isnecessary to know the methods by whichparticipants were allocated to interventiongroups. 18 Specifically, in order to assess the degreeof selection bias we need to know the factorsinfluencing the allocation, the manner in whichthose factors are related to outcome and how thefactors were dealt with in the analysis. Theserequirements underlie our choice of the fourcore items we propose for inclusion in qualityassessment tools for non-randomised studies(Chapter 4), although we found that theywere rarely included in the tools that weassessed.In classical non-randomised designs, allocation isoften made according to centre (concurrentcontrolled cohort studies) or period (historicallycontrolled cohort studies). However, knowing thatthere is a difference in location or era of thetreated and control samples is inadequate to judgecomparability: it is necessary to know the mannerin which these differences lead to specificdifferences in case-mix.The situation is more complex when interventionsbeing compared have been used across centresduring the same period, where allocation is byindication or patient preference. Then it is notusually possible to describe fully the factorsinfluencing allocation, nor is it likely that theallocation mechanism will be consistent acrosspatients and clinicians. In addition, patients maybe included in a study who are suitable for onlyone of the two treatments. These factors all meanthat allocation may be strongly linked toprognostic factors, such that the bias related todifferences in case-mix will be large. 3Theoretically, the second stage in assessing thelikelihood of bias in a non-randomised study is toconsider whether differences in confoundingfactors are comparable between the groups and, ifnot, whether they have been adjusted for in theanalysis. Many investigators routinely claim effectsas ‘adjusted’ or ‘independent’ or ‘unbiased’ whenthey have used statistical methods to correct fordifferences in case-mix. However, their trust in theobserved comparability of case-mix and the abilityof statistical adjustment methods to adjustproperly for differences in case-mix is unfounded.The unique design of our investigation hasallowed the performance of these case-mixadjustment methods to be assessed by comparingunadjusted and adjusted results with those fromrandomised comparisons. We have onlyundertaken these evaluations in two data sets andwe found consistent results indicating that casemixadjustment cannot reliably compensate forbiases in selection, and in fact it may introduceadditional bias.89© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Discussion and conclusions90For example, the biases observed in our empiricalinvestigations based on cohort designs could notbe removed using case-mix adjustment methods,even though we could describe the process bywhich allocation occurred (Chapter 7). This wasdespite having at our disposal complete data onseveral highly prognostic variables. Our principalfailure resulted from being unable to describe howallocation was linked to prognostic factors in amanner that could be adjusted for in a statisticalanalysis. Moses 18 stated that there are threerequirements for successful case-mix adjustment:(1) knowledge of which variables must be takeninto account, (2) measuring those variables on eachparticipant and (3) using those measurementsappropriately to adjust the treatment comparison.He suggested that researchers are likely to fail at allthree. Although it is well known that failure toidentify all relevant variables will lead to underadjustment(residual confounding), our evaluationshave suggested that failing to measure thecorrect variables properly and failing to modeltheir effects appropriately can lead to spuriousconclusions.The use of case-mix adjustment analyses shouldtherefore not be regarded as a guarantee that astudy is unbiased, or that the observed differencesin case-mix have been adjusted for. Investigatorsusing case-mix adjustment methodologies,especially logistic regression models, should drawconclusions cautiously, and probably regardadjusted analyses as exploratory rather thanconfirmatory. Our observations require furtherassessment and confirmation from investigationsin additional data sets to explore further themechanisms that are causing these failures.Nonetheless, they raise serious concern about theroutine promotion and use of case-mix adjustmentmethods for the analysis of non-randomisedstudies. We do not propose that case-mixadjustment be abandoned, but recommend thatgreater scepticism be applied in the interpretationof results.What are the implications for thoseproducing, reviewing and using nonrandomisedstudies?An investigator planning to undertake a nonrandomisedstudy should first make certain thatan RCT cannot be undertaken. 18 The advantagesof concealed randomised allocation are so greatcompared with the inadequacies that we havequantified for non-randomised studies that itshould be with the greatest reluctance that aninvestigator concludes that randomisation is notpossible. The ability to eradicate bias at thedesign stage is crucial to establishing thevalidity of a study. In particular, investigatorsshould not assume that statistical methods canbe used reliably to compensate for biasesintroduced through suboptimal allocationmethods.A prospective non-randomised study should beundertaken according to a protocol that iscarefully followed to ensure consistent inclusioncriteria, that all relevant factors are measuredaccurately for each participant and thatparticipants are all monitored in a standardmanner and blinded to treatment if possible. Insome situations it may even be possible to matchprospectively treated and control patients onimportant prognostic factors. 171 Byar pointed outthat the one piece of information required forsuccessful adjustment is knowledge of the reasonswhy each patient received their particulartreatment. 16 This information is usually notrecorded in non-randomised studies, and how itshould be measured is a real challenge worthy ofresearch. 18 As a minimum, recording details ofreasons for allocations of particular interventionsshould allow the subset of patients to be identifiedwho are not considered suitable for bothtreatments, and should not be included inanalyses. In addition, all prognostic variablesshould be measured in such a way as to allowmeasurement error, misclassification and withinpersonvariability to be assessed. Some authorshave attempted to acquire reliable risk assessmentsby retrospective case-note review, 172 but obtainingthese data prospectively has many attractions.Retrospective studies cannot use consistentinclusion criteria, or ensure that data arecomplete and consistently recorded. Phillipsand Davey Smith have pointed out that it isprobably more worthwhile to put effort intoundertaking a small observational study to ahigh standard than in obtaining poor qualityand inadequate data on a large number ofparticipants. 173 However, despite all these efforts,the possibility of residual confounding will not beremoved.Several authors have suggested additionalstrategies that investigators might consideras a way of identifying hidden bias. Rosenbaumproposed the routine use of extra controlgroups and the inclusion of additionaloutcomes that are known not to be alteredby the treatment as further checks oncomparability. 145,152 Several authors haveemphasised the use of sensitivity analyses toquantify the strength of confounding in a missing

Health Technology Assessment 2003; Vol. 7: No. 27covariate that would be needed to nullify anobserved effect. 145,174 However, none of thesemethods can correct for bias.Investigators undertaking systematic reviews ofeffectiveness should include the results of nonrandomisedstudies with discretion. If results fromgood-quality RCTs are available, there seems to beno justification for additionally considering nonrandomisedstudies. If non-randomised studies areto be included in a review, their quality needs tobe carefully assessed to evaluate the likelihood ofbias. Quality assessment should pay particularattention to the description of the allocationmechanisms and the demonstration of there beingcomparability in all important prognostic factorsat baseline. Although we have not identified aquality assessment tool that met all ourrequirements, the six best tools could all beadapted for use in a systematic review. Theconclusions of a review should take into accountthe extra uncertainty associated with the results ofnon-randomised studies.The results of non-randomised studies should betreated with a healthy degree of scepticism.Healthcare decision-makers should be cautious notto over-interpret results from non-randomisedstudies. Importantly, checking that treated andcontrol groups appear comparable does notguarantee freedom from bias, and it should neverbe assumed that case-mix adjustment methods canfully correct for observed differences betweengroups. The uncertainty in the result of a nonrandomisedstudy is not properly summarised inthe confidence interval for the overall effect: ouranalyses have shown that the true uncertainty inthe results of non-randomised studies may be 10times greater.ConclusionsNon-randomised studies are sometimes but notalways biased:●●The results of non-randomised studies can differfrom the results of RCTs of the same intervention.All other issues remaining equal, lack ofrandomisation introduces bias into theassessment of treatment effects. The bias mayhave two components. It may be systematic andappear on average to act in a particulardirection if the non-random allocationmechanism leads to a consistent difference incase-mix. However, if the allocation mechanismcan lead to haphazard differences in case-mix,●the bias can act in either direction, increasinguncertainty in outcome in ways that cannot bepredicted. The extent of systematic bias andincreased uncertainty varies according to thetype of non-randomised comparison andclinical context.Meta-epidemiological techniques tend not toprovide useful information on the sources ofand degrees of bias in non-randomised studiesowing to the existence of meta-confounding andlack of systematic or predictable bias in theresults of non-randomised studies.Statistical methods of analysis cannot properlycorrect for inadequacies of study design:●Case-mix comparability and standard methodsof case-mix adjustment do not guarantee theremoval of bias. Residual confounding may behigh even when good prognostic data areavailable, and in some situations adjustedresults may appear more biased thanunadjusted results.Systematic reviews of effectiveness often do notadequately assess the quality of non-randomisedstudies:●●Quality assessment has not routinely beenundertaken in systematic reviews of effectivenessthat include non-randomised studies. Whenstudy quality has been investigated, there isvariability in the tools and quality criteria thathave been used and there has been no consistentpattern between quality and review findings.Not all reviews that have assessed quality haveconsidered the findings when synthesising studyresults and drawing conclusions.Although many quality assessment tools existand have been used for appraising nonrandomisedstudies, few are suitable for thistask as they omit key domains of qualityassessment (assessment of allocationmechanisms, attempts to achieve case-mixcomparability by design, identification ofconfounding factors, use of case-mix adjustmentmethods). Few were developed usingappropriate methodological procedures.Fourteen tools were identified which appear tohave reasonable coverage of core domains ofinternal validity, six of which were consideredpotentially suitable for use in systematic reviewsof non-randomised studies. All six wouldrequire modification to cover adequately the keyissues in non-randomised studies (of identifyingprognostic factors and accounting for them inthe design and analysis).91© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Discussion and conclusionsNon-randomised studies provide a poor basis fortreatment or health policy decisions:●●The inability of case-mix adjustment methodsto compensate for selection bias and ourinability to identify non-randomised studieswhich are free of selection bias indicate thatnon-randomised studies should only be used insituations where RCTs cannot be undertaken.Healthcare policies based upon nonrandomisedstudies or systematic reviews ofnon-randomised studies may need re-evaluationif the uncertainty in the true evidence base wasnot fully appreciated when the decisions weremade.Recommendations for further research1. The resampling methodology that we haveemployed in this project should be applied inother clinical areas where suitable RCTs exist.These evaluations should consider (a) thedistribution of biases associated with nonrandomisedallocation, (b) whether nonrandomisedstudies with similar baselinecharacteristics are less biased and (c) theperformance of case-mix adjustment methods.It would be valuable to study different contextsto evaluate the degree to which bias is relatedto the amount of prognostic information knownat allocation.2. Efforts should be focused on the developmentof a new quality assessment tool for nonrandomisedstudies or the refinement anddevelopment of existing tools. Appropriatemethodological procedures of tooldevelopment should be employed and keyindicators of internal validity covered. Theseindicators include both those for whichempirical evidence is available from work onRCTs and those supported by our empiricalinvestigations. The latter should include themethod used to allocate participants to groups;specification of the factors that influenced theseallocations; the way in which these factors arethought to relate to outcome; and appropriateadjustment in the analysis. In the meantime,systematic reviewers should be stronglyencouraged to use and adapt those tools thatdo cover key quality issues.3. Research should be undertaken to developmethods of measuring and characterisingreasons for treatment choices in patientpreference and allocation by indication studies,and evaluations undertaken to assess whetherrecording such information allows effectiveadjustment for selection bias.4. Empirical work is needed to investigatehow quality assessments of non-randomisedstudies should be incorporated in thesynthesis of studies in a systematic reviewand to study the implications of individualquality features for the interpretation of reviewresults.5. Reasons for the failure of case-mix adjustmentmethods should be further investigated,including assessment of the generalisability ofour results to risk assessments andepidemiological studies where they arefrequently utilised. The impact of differencesbetween unconditional and conditionalestimates of treatment effects should beassessed.6. Guidelines should be produced to adviseinvestigators on the best ways of undertakingprospective non-randomised studies tominimise bias.7. The role of propensity scoring in adjusting forselection bias should be further evaluated, andcomputer macros made available for itsapplication.Recommendations for those producingand using health technologyassessments1. Systematic reviewers and those conductinghealth technology assessments should bestrongly encouraged to base any estimates ofeffectiveness or cost-effectiveness on evidencefrom RCTs. Where such evidence is unavailable,the uncertainty inherent in estimates based onnon-randomised evidence should be stronglyemphasised.2. Decision-makers should review healthcarepolicies based on the results of non-randomisedstudies to assess whether the inherentuncertainty in the evidence base was properlyappreciated when the policy decisions weremade.3. Agencies funding primary research should fundnon-randomised studies only when they areconvinced that a randomised study is notfeasible.92

Health Technology Assessment 2003; Vol. 7: No. 27AcknowledgementsWe would like to thank the following peoplefor their help with this project: PeterSandercock and Peter Rothwell for organisingaccess to the IST and ECST data setsrespectively; Jane Harrison for carrying outthe extensive set of searches and Kath Wrightfor help with search queries; Alison Brown andVanda Castle for secretarial support. We wouldalso like to acknowledge the helpful commentsof the members of our expert panel:Professor Brian Haynes, Professor Nick Black andDr Barnaby Reeves.93© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Health Technology Assessment 2003; Vol. 7: No. 27References1. D’Agostino RB, Kwan H. Measuring effectiveness:what to expect without a randomized controlgroup. Med Care 1995;33:95–105.2. Abel U, Koch A. The role of randomization inclinical studies: myths and beliefs. J Clin Epidemiol1999;52:487–97.3. Green SB, Byar DP. Using observational data fromregistries to compare treatments: the fallacy ofomnimetrics. Stat Med 1984;3:361–73.4. Chalmers I. Assembling comparison groups toassess the effects of health care. J R Soc Med1997;90:379–86.5. Black N. Why we need observational studies toevaluate the effectiveness of health care. BMJ1996;312:1215–18.6. Altman DG, Schulz KF. Statistics notes –concealing treatment allocation in randomisedtrials. BMJ 2001;323:446–7.7. Feinstein AR, Horwitz RI. Problems in the“evidence” of “evidence-based medicine”. Am JMed 1997;103:529–35.8. Bradley C. Designing medical and educationalintervention studies: a review of some alternativesto conventional randomized controlled trials.Diabetes Care 1993;16:509–18.9. Torgerson D, Klaber-Moffett J, Russell I. Patientpreferences in randomised trials: threat oropportunity. J Health Serv Res Policy 1996;1:194–7.10. Bradford Hill A. A short textbook of medicalstatistics. 10th ed. London: Hodder & Stoughton,1977.11. Miettinen OS. Theoretical epidemiology:principles of occurrence research in medicine.New York: Wiley, 1985.12. Greenland S, Neutra R. Control of confounding inthe assessment of medical technology. Int JEpidemiol 1980;9:361–7.13. Salas M, Hofman A, Stricker BH. Confounding byindication: an example of variation in the use ofepidemiologic terminology. Am J Epidemiol1999;149:981–3.14. Walker AM. Confounding by indication.Epidemiology 1996;7:335–6.15. Horwitz RI, Feinstein AR. The application oftherapeutic trial principles to improve the designof epidemiologic research. J Chron Dis1981;34:575–83.16. Byar DP. Problems with using observational databases to compare treatments. Stat Med1991;10:663–6.17. Horwitz RI, Viscoli CM, Clemens JD, Sadock RT.Developing improved observational methods forevaluating therapeutic effectiveness. Am J Med1990;89:630–8.18. Moses LE. Measuring effects without randomizedtrials? Options, problems, challenges. Med Care1995;33:AS8–AS14.19. Altman DG, Schulz KF, Moher D, Egger M,Davidoff F, Elbourne D, et al. The revisedCONSORT statement for reporting randomizedtrials: explanation and elaboration. Ann Intern Med2001;134:663–94.20. Schulz KF, Chalmers I, Hayes RJ, Altman DG.Empirical evidence of bias. Dimensions ofmethodological quality associated with estimates oftreatment effects in controlled trials. JAMA1995;273:408–12.21. Moher D, Pham B, Jones A, Cook DJ, Jadad AR,Moher M, et al. Does quality of reports ofrandomised trials affect estimates of interventionefficacy reported in meta-analyses? Lancet1998;352:609–13.22. Sterne JAC, Jüni P, Schulz KF, Altman DG, BartlettC, Egger M. Statistical methods for assessing theinfluence of study characteristics on treatmenteffects in ‘meta-epidemiological’ research. Stat Med2002;21:1513–24.23. Jüni P, Altman DG, Egger M. Assessing thequality of randomised controlled trials. InEgger M, Davey Smith G, Altman DG, editors.Systematic reviews in health care: meta-analysisin context. London: BMJ Books; 2001.pp. 87–108.24. Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ,Prins MH, van der Meulen JH, et al. Empiricalevidence of design-related bias in studies ofdiagnostic tests. JAMA 1999;282:1061–6.25. Britton A, McKee M, Black N, McPherson K,Sanderson C, Bain C. Choosing betweenrandomised and non-randomised studies: asystematic review. Health Technol Assess1998;2(13).26. MacLehose RR, Reeves BC, Harvey IM, SheldonTA, Russell IT, Black AMS. A systematic review ofcomparisons of effect sizes derived fromrandomised and non-randomised studies. HealthTechnol Assess 2000;4(34).95© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

References9627. Sacks H, Chalmers TC, Smith H Jr. Randomizedversus historical controls for clinical trials. Am JMed 1982;72:233–40.28. Kunz R, Oxman AD. The unpredictabilityparadox: review of empirical comparisons ofrandomised and non-randomised clinical trials.BMJ 1998;317:1185–90.29. Kunz R, Vist G, Oxman AD. Randomisation toprotect against selection bias in healthcare trials(Cochrane Methodology Review). In TheCochrane Library, Issue 4. Oxford: UpdateSoftware; 2002.30. Chalmers TC, Matta RJ, Smith HJ. Evidencefavoring the use of anticoagulants in the hospitalphase of acute myocardial infarction. N Engl J Med1982;72:233–40.31. Diehl LF, Perry DJ. A comparison of randomizedconcurrent control groups with matched historicalcontrol groups: are historical controls valid? J ClinOncol 1986;4:1114–20.32. Benson K, Hartz A. A comparison of observationalstudies and randomized, controlled trials. N Engl JMed 2000;342:1878–86.33. Concato J, Shah N, Horwitz RI. Randomized,controlled trials, observational studies, and thehierarchy of research designs. N Engl J Med2000;342:1887–92.34. Ioannidis JPA, Haidich A, Pappa M, Pantazis N,Kokori SI, Tektonidou MG, et al. Comparison ofevidence of treatment effects in randomized andnon randomized studies. JAMA 2001;286:821–30.35. Lipsey MW, Wilson DB. The efficacy ofpsychological educational, and behaviouraltreatments: confirmation from meta-analysis. AmPsychol 1993;48:1181–209.36. Wilson DB, Lipsey MW. The role of method intreatment effectiveness research: evidence frommeta-analysis. Psychol Methods 2001;6:413–29.37. Miller JN, Colditz GA, Mosteller F. How studydesign affects outcomes in comparisons of therapy.II: surgical. Stat Med 1989;8:455–66.38. Colditz GA, Miller JN, Mosteller F. How studydesign affects outcomes in comparisons of therapy.I: medical. Stat Med 1989;8:441–54.39. Ottenbacher K. Impact of random assignment onstudy outcome: an empirical examination. ControlClin Trials 1992;13:50–61.40. Pocock SJ, Elbourne DR. Randomized trials orobservational tribulations? N Engl J Med2000;342:1907–9.41. Kunz R, Khan KS, Nuemayer H, Sacks HS,Liu P-Y, Anderson G, et al. Observational studiesand randomized trials. N Engl J Med 2000;343:1194–7.42. Dickersin K. How important is publication bias?A synthesis of available data. AIDS Educ Prev 1997;9:15–21.43. Kjaergard LL, Villumsen J, Gluud C. Reportedmethodologic quality and discrepancies betweenlarge and small randomized trials in metaanalyses.Ann Intern Med 2001;135:982–9.44. Moher D, Jadad AR, Nichol G, Penman M, TugwellP, Walsh S. Assessing the quality of randomizedcontrolled trials: an annotated bibliography of scalesand checklists. Control Clin Trials 1995;16:62–73.45. Cooper H. The integrative research review: asystematic approach. Newbury Park, CA: Sage; 1984.46. Jadad AR, Moore RA, Carroll D, Jenkinson C,Reynolds DJ, Gavaghan DJ, et al. Assessing thequality of reports of randomized clinical trials: isblinding necessary? Control Clin Trials 1996;17:1–12.47. Clark HD, Wells GA, Huet C, McAlister FA, SalmiLR, Fergusson D, et al. Assessing the quality ofrandomized trials: reliability of the Jadad scale.Control Clin Trials 1999;20:448–52.48. Wortman PM. Judging research quality. In CooperH, Hedges LV, editors. The handbook of researchsynthesis. New York: Russell Sage Foundation;1994. pp. 97–109.49. Campbell DT, Stanley JC, editors. Experimentaland quasi-experimental designs for research.Chicago, IL: Rand McNally; 1966.50. Cook TD, Campbell DT. Quasi-experimentation:design and analysis issues for field settings.Chicago, IL: Rand McNally; 1979.51. Chalmers TC, Smith H, Blackburn B, SilvermanB, Schroeder B, Reitman D, et al. A method forassessing the quality of a randomized control trial.Control Clin Trials 1981;2:31–49.52. Bangert-Drowns RL, Wells-Parker E, Chevillard I.Assessing the methodological quality of research innarrative reviews and meta-analyses. In Bryant KJ,Windle M, editors. The science of prevention:methodological advances from alcohol andsubstance abuse research. Washington, DC:American Psychological Association; 1997.pp. 405–29.53. Moher D, Jadad AR, Tugwell P. Assessing thequality of randomized controlled trials. Currentissues and future directions. Int J Technol AssessHealth Care 1996;12:195–208.54. Jüni P, Altman DG, Egger M. Systematic reviews inhealth care: assessing the quality of controlledclinical trials. BMJ 2001;323:42.55. Schulz KF, Chalmers I, Altman DG, Grimes DA,Dore CJ. The methodologic quality ofrandomization as assessed from reports of trials inspecialist and general medical journals. Online JCurr Clin Trials 1995;Aug 26:Doc No 197.

Health Technology Assessment 2003; Vol. 7: No. 2756. Chalmers TC, Celano P, Sacks HS, Smith H Jr.Bias in treatment assignment in controlled clinicaltrials. N Engl J Med 1983;309:1358–61.57. Streiner DL, Norman GR. Health measurementscales: a practical guide to their development anduse. Oxford: Oxford University Press; 1995.58. Greenland S. Invited commentary: a critical lookat some popular meta-analytic methods. Am JEpidemiol 1994;140:290–6.59. Jüni P, Witschi A, Bloch R, Egger M. The hazardsof scoring the quality of clinical trials for metaanalysis.JAMA 1999;282:1054–60.60. Brazier J, Deverill M, Green C, Harper R, BoothA. A review of the use of health status measures ineconomic evaluation. Health Technol Assess 1999;3(9).61. Morris JJ, Smith R, Glower DD, Muhlbaier LH,Reves JG, Wechsler AS, et al. Clinical evaluation ofsingle versus multiple mammary artery bypass.Circulation 1990;82:214–23.62. Nichol K, Baken L, Wuorenma J, Nelson A. Thehealth and economic benefits associated withpneumococcal vaccination of elderly persons withchronic lung disease. Arch Intern Med1999;159:2437–42.63. Walker JD, Dodds RA, Murrells TJ, Bending JJ,Mattock MB, Keen H. Restriction of dietaryprotein and progression of renal failure in diabeticnephropathy. Lancet 1989;ii:1411–14.64. Critical Appraisal Skills Programme. 12 questionsto help you make sense of a cohort study. Oxford:Critical Appraisal Skills Programme; 1999.65. Thomas H. Quality assessment tool forquantitative studies. Effective Public HealthPractice Project. McMaster University,Toronto.66. Wells G, Shay B. Data extraction for nonrandomisedsystematic reviews. University ofOttawa, Ottawa.67. Verhagen AP, de Vet HC, de Bie RA, Kessels AG,Boers M, Knipschild PG. Balneotherapy andquality assessment: interobserver reliability of theMaastricht criteria list and the need for blindedquality assessment. J Clin Epidemiol 1998;51:335–41.68. ter Riet G, Kleijnen J, Knipschild P. Acupunctureand chronic pain: a criteria-based meta-analysis.J Clin Epidemiol 1990;43:1191–9.69. Kleijnen J, Knipschild P, Ter Riet G. Clinical trialsof homeopathy. BMJ 1991;302:316–32.70. Koes BW, Assendelft WJ, van der Heijden GJ,Bouter LM, Knipschild PG. Spinal manipulationand mobilisation for back and neck pain: ablinded review. BMJ 1991;303:1298–303.71. van der Windt DA, van der Heijden GJ,Scholten RJ, Koes BW, Bouter LM. The efficacy ofnon-steroidal anti-inflammatory drugs forshoulder complaints. J Clin Epidemiol1995;48:691–704.72. de Vet HC, de Bie RA, van der Heijden GJ,Verhagen AP, Sijpkes P, Knipschild PG. Systematicreviews on the basis of methodologic criteria.Physiotherapy 1997;83:284–9.73. Bours GJ, Ketelaars CA, Frederiks CM, Abu SaadHH, Wouters EF. The effects of aftercare onchronic patients and frail elderly patients whendischarged from hospital: a systematic review.J Adv Nurs 1998;27:1076–86.74. Kleijnen J, Riet ter G, Knipschild P. Vitamine C enverkoudheid. Overzicht van een megadosisliteratuur. Ned Tijdschr Geneeskd 1989;133:1535.75. Riet ter G, Kleijnen J, Knipschild P. De metaanalyseals review-methode [De effectiviteit vanacupunctuur]. Huisarts en Wetenschap 1989;32:176–81.76. Antczak AA, Tang J, Chalmers TC. Qualityassessment of randomized control trials in dentalresearch 1. Methods. J Periodontal Res 1986;21:305–14.77. Nicolucci A, Grilli R, Alexanian A, Apolone G,Torri V, Liberati A. Quality, evolution, and clinicalimplications of randomized, controlled trials onthe treatment of lung cancer. JAMA 1989;262:2101–7.78. Guyatt GH, Sackett DL, Cook DJ. Users’ guides tothe medical literature. II. How to use an articleabout therapy or prevention. B. What were theresults and will they help me in caring for mypatients? Evidence-based Medicine WorkingGroup. JAMA 1994;271:59–63.79. Guyatt GH, Sackett DL, Cook DJ. Users’ guides tothe medical literature. II. How to use an articleabout therapy or prevention. A. Are the results ofthe study valid? Evidence-based Medicine WorkingGroup. JAMA 1993;270:2598–601.80. Department of Clinical Epidemiology andBiostatistics, McMaster University Health SciencesCentre. How to read clinical journals. V. Todistinguish useful from useless or even harmfultherapy. CMAJ 1981;124:1156–62.81. Anders JF, Jacobson RM, Poland GA, Jacobsen SJ,Wollan PC. Secondary failure rates of measlesvaccines: a meta-analysis of published studies.Pediatr Infect Dis J 1996;15:62–6.82. Baker DW, Jones R, Hodges J, Massie BM,Konstam MA, Rose EA. Management of heartfailure. III. The role of revascularization in thetreatment of patients with moderate or severe leftventricular systolic dysfunction. JAMA1994;272:1528–34.97© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

References9883. Cameron I, Crotty M, Currie C, Finnegan T,Gillespie L, Gillespie W, et al. Geriatricrehabilitation following fractures in older people:a systematic review. Health Technol Assess 2000;4(2).84. Cho MK, Bero LA. Instruments for assessing thequality of drug studies published in the medicalliterature. JAMA 1994;272:101–4.85. Downs SH, Black N. The feasibility of creating achecklist for the assessment of the methodologicalquality both of randomised and non-randomisedstudies of health care interventions. J EpidemiolCommun Health 1998;52:377–84.86. Zaza S, Wright-de Aguero LK, Briss PA, TrumanBI, Hopkins DP, Hennessy MH, et al. Datacollection instrument and procedure for systematicreviews in the ‘Guide to Community PreventiveServices’. Am J Prev Med 2000;18:44–74.87. Miller WR, Brown JM, Simpson TL. What works?A methodological analysis of the alcohol treatmentoutcome literature. In Hester RK, Miller WR,editors. Handbook of alcoholism treatmentapproaches: effective alternatives. Boston, MA:Allyn & Bacon; 1995. pp. 12–44.88. Bass JL, Christoffel KK, Widome M, Boyle W,Scheidt P, Stanwick R, et al. Childhood injuryprevention counseling in primary care settings: acritical review of the literature. Pediatrics 1993;92:544–50.89. Duffy ME. A research appraisal checklist forevaluating nursing research reports. Nurs HealthCare 1985;6:538–47.90. Shay CB, Zimmermann WS. The factorial validityof a rating scale for the evaluation of researcharticles. Educ Psychol Meas 1972;32:453–7.91. Wells-Parker E, Bangert-Drowns R. Meta-analysisof research on DUI remedial interventions. DrugsAlcohol Driving 1990;6:147–60.92. Detsky AS, Naylor CD, O’Rourke K, McGeer AJ,L’Abbe KA. Incorporating variations in the qualityof individual randomized trials into meta analysis.J Clin Epidemiol 1992;45:255–65.93. Melchart D, Linde K, Worku F, Bauer R, WagnerH. Immunomodulation with Echinacea – asystematic review of controlled clinical trials.Phytomedicine 1994;1:245–54.94. Standards of Reporting Trials Group. A proposalfor structured reporting of randomised controlledtrials. JAMA 1994;272:1926–31.95. Linde K, Scholz M, Ramirez G, Clausius N,Melchart D, Jonas WB. Impact of study quality onoutcome in placebo-controlled trials ofhomeopathy. J Clin Epidemiol 1999;52:631–6.96. de Oliveira IR, Dardennes RM, Amorim ES,Diquet B, de Sena EP, Moreira EC, et al. Is there arelationship between antipsychotic blood levelsand their clinical efficacy? An analysis of studiesdesign and methodology. Fundam Clin Pharmacol1995;9:488–502.97. Coleridge Smith P. The management of chronicvenous disorders of the leg: an evidence-basedreport of an International Task Force. Phlebology1999;14:3–19.98. Dawson-Saunders B, Trapp R. Reading themedical literature. Basic and clinical biostatistics.Norwalk, CT: Appleton & Lange; 1990.pp. 264–76.99. DuRant RH. Checklist for the evaluation ofresearch articles. J Adolesc Health 1994;15:4–8.100. Friedenreich CM. Methods for pooled analyses ofepidemiologic studies. Epidemiology 1993;4:295–302.101. Gordis L, Kleinman JC, Klerman LV, Mullen PD,Paneth N. Criteria for evaluating evidenceregarding the effectiveness of prenatalinterventions. In Merkatz IR, Thompson JE,editors. New perspectives on prenatal care. NewYork: Elsevier; 1990. pp. 31–8.102. Hadorn DC, Baker D, Hodges JS, Hicks N. Ratingthe quality of evidence for clinical practiceguidelines. J Clin Epidemiol 1996;49:749–54.103. Wilson A, Henry DA. Meta-analysis. Part 2.Assessing the quality of published meta-analyses.Med J Aust 1992;156:173–87.104. Bracken MB. Reporting observational studies. Br JObstet Gynaecol 1989;96:383–8.105. Spitzer WO, Lawrence V, Dales R, Hill G, ArcherMC, Clarck P, et al. Links between passive smokingand disease; a best-evidence synthesis. Clin InvestMed 1990;13:17–42.106. Talley NJ, Nyren O, Drossman DA. The irritablebowel syndrome: toward optimal design ofcontrolled treatment trials. Gastroenterology Int1993;6:189–211.107. Fowkes FG, Fulton PM. Critical appraisal ofpublished research. Introductory guidelines. BMJ1991;302:1136–40.108. Weintraub M. How to critically assess clinical drugtrials. Drug Ther 1982;12:131–48.109. Cowley DE. Prostheses for primary total hipreplacement. A critical appraisal of the literature.Int J Technol Assess Health Care 1995;11:770–8.110. Vickers A. Critical appraisal: how to read a clinicalresearch paper. Complement Ther Med 1995;3:158–66.111. Reisch J, Tyson JE, Mize SG. Aid to the evaluationof therapeutic studies. Pediatrics 1989;84:815–27.112. Downs, Black N. Systematic review of the literatureon the effectiveness of surgery for stressincontinence for women. London: London Schoolof Hygiene and Tropical Medicine; 1996.

Health Technology Assessment 2003; Vol. 7: No. 27113. Taggart DP, D’Amico R, Altman DG. Effect ofarterial revascularisation on survival: a systematicreview of studies comparing bilateral and singleinternal mammary arteries. Lancet 2001;358:870–5.114. Tyson JE, Furzan JA, Reisch JS, Mize SG. Anevaluation of the quality of therapeutic studies inperinatal medicine. Obstet Gynecol 1983;62:99–102.115. McAuley L, Pham B, Tugwell P, Moher D. Doesthe inclusion of grey literature influence estimatesof intervention effectiveness reported in metaanalyses?Lancet 2000;356:1228–31.116. Khan KS, Daya S, Jadad A. The importance ofquality of primary studies in producing unbiasedsystematic reviews. Arch Intern Med1996;156:661–6.117. Stroup DF, Berlin JA, Morton SC, Olkin I,Williamson GD, Rennie D, et al. Meta-analysis ofobservational studies in epidemiology: a proposalfor reporting. Meta-analysis of ObservationalStudies in Epidemiology (MOOSE) group. JAMA2000;283:2008–12.118. Clarke M, Oxman AD, editors. CochraneReviewers Handbook 4.1.5 [updated April 2002].In The Cochrane Library, Issue 2. Oxford: UpdateSoftware; 2002.119. NHS Centre for Reviews and Dissemination.Undertaking systematic reviews of research oneffectiveness. CRD Report No. 4. York: Universityof York, 1996.120. L’Abbe KA, Detsky AS, O’Rourke KF. Meta-analysisin clinical research. Ann Intern Med 1987;107:224–33.121. Schmid JE, Koch GC, LaVange LM. An overviewof statistical issues and methods of meta-analysis.J Biopharm Stat 1991;1:103–20.122. Slavin RE. Best evidence synthesis: an intelligentalternative to meta-analysis. J Clin Epidemiol 1995;48:9–18.123. Ganong LH. Integrative reviews of nursingresearch. Res Nurs Health 1987;10:1–11.124. Gartland JJ. Orthopaedic clinical research:deficiencies in experimental design anddeterminations of outcome. J Bone Joint Surg Am1988;70:309–15.125. Smith TMF, Sugden RA. Sampling and assignmentmechanisms in experiments, surveys andobservational studies. Int Stat Rev 1988;56:165–80.126. Sackett DL. Rules of evidence and clinicalrecommendations on the use of antithromboticagents. Chest 1989;95:2s–4s.127. Gyorkos TW, Tannenbaum TN, Abrahamowicz M,Oxman AD, Scott EA, Millson ME, et al. Anapproach to the development of practiceguidelines for community health interventions.Can J Public Health 1994;85:s8–s13.128. Oakley A, Fullerton D, Holland J, Arnold S,Hickey D, Kelley P, et al. SSRU database project:reviews of effectiveness: HIV prevention andsexual health education interventions. London:University of London, Social Science ResearchUnit; 1994.129. Wells-Parker E, Bangert-Drowns R, McMillen R,Williams M. Final results from a meta-analysis ofremedial interventions with drink/drive offenders.Addiction 1995;90:907–26.130. Cheek CM, Black NA, Devlin HB, Kingsnorth AN,Taylor RS, Watkin DF. Groin hernia surgery: asystematic review. Ann R Coll Surg Engl 1998;80:S1–S80.131. Moher D, Cook DJ, Jadad AR, Tugwell P, MoherM, Jones A, et al. Assessing the quality of reportsof randomised trials: implications for the conductof meta-analyses. Health Technol Assess 1999;3(12).132. International Stroke Trial Collaborative Group.The International Stroke Trial (IST): a randomisedtrial of aspirin, subcutaneous heparin, both, orneither among 19435 patients with ischaemicstroke. Lancet 1997;349:1569–81.133. European Carotid Surgery Trialists’ CollaborativeGroup. Randomised trial of endarterectomy forrecently symptomatic carotid stenosis: final resultsof the MRC European Cartoid Surgery Trial(ECST). Lancet 1998;351:1379–87.134. Efron B, Tibshirani RJ. An introduction to thebootstrap. London: Chapman & Hall; 1993.135. North American Symptomatic CarotidEndarterectomy Trial (NASCET) investigators.Clinical alert: benefit of carotid endarterectomyfor patients with high-grade stenosis of theinternal carotid artery. National Institute ofNeurological Disorders and Stroke, Stroke andTrauma Division. Stroke 1991;22:816–17.136. Iezzoni LI. Risk adjustment for measurementhealthcare outcomes. 2nd ed. Chicago, IL: HealthAdministration Press; 1997.137. Sowden AJ, Deeks J, Sheldon TA. Volume andoutcome in coronary artery bypass graft surgery:true relationship or artefact? BMJ1995;311:151–5.138. Naylor CD, Guyatt GH. Therapy and harm –outcomes of health services. In Guyatt GH,Rennie D, editors. Users’ guides to the medicalliterature – a manual for evidence-based clinicalpractice. Chicago, IL: American MedicalAssociation Press; 2002. pp. 233–46.139. Levine M, Haslam D, Walter S, Cumming R,Lee H, Haines T, et al. Harm. In Guyatt GH,Rennie D, editors. Users’ guides to the medicalliterature – a manual for evidence-based clinical99© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

References100practice. Chicago, IL: American MedicalAssociation Press; 2002. pp. 84–100.140. Altman DG. Practical statistics for medicalresearch. London: Chapman & Hall; 1991.141. Breslow NE, Day NE, editors. Statistical methodsin cancer research. Analysis of case control studies.Lyon: IARC Scientific Publications; 1980.142. Breslow NE, Day NE. Statistical methods in cancerresearch. Design and analysis of cohort studies.Lyon: IARC Scientific Publications; 1987.143. Altman DG. Adjustment for covariate imbalance.In Armitage P, Colton T, editors. Encyclopedia ofbiostatistics. Chichester: Wiley; 1998. pp. 1005–7.144. Mantel N, Haenszel W. Statistical aspects of theanalysis of data from retrospective studies ofdisease. J Natl Cancer Inst 1959;22:719–48.145. Rosenbaum PR. Observational studies. New York:Springer; 1995.146. Rosenbaum PR, Rubin DB. The central role of thepropensity score in observational studies for causaleffects. Biometrica 1983;70:41–55.147. D’Agostino RB. Propensity score methods for biasreduction in the comparison of a treatment to anon-randomized control group. Stat Med 1998;17:2265–81.148. Robins J, Greenland S, Breslow NE. A generalestimator for the variance of the Mantel–Haenszelodds ratio. Am J Epidemiol 1986;124:719–23.149. Rothman KJ. Modern epidemiology. Boston, MA:Little Brown; 1986.150. Leon DA. Failed or misleading adjustment forconfounding. Lancet 1993;342:479–81.151. Robinson LD, Jewell NP. Some surprising resultsabout covariate adjustment in logistic-regressionmodels. Int Stat Rev 1991;59:227–40.152. Rosenbaum PR. Discussing hidden bias inobservational studies. Ann Intern Med 1991;115:901–5.153. Cochran WG. The effectiveness of adjustment bysubclassification in removing bias in observationalstudies. Biometrics 1968;24:295–313.154. Brenner H, Blettner M. Controlling for continuousconfounders in epidemiologic research.Epidemiology 1997;8:429–34.155. Brenner H. A potential pitfall in control ofcovariates in epidemiologic studies. Epidemiology1997;9:68–71.156. Davey Smith G, Phillips A. Declaringindependence: why we should be cautious.J Epidemiol Community Health 1990;44:257–8.157. Greenland S. The effect of misclassification in thepresence of covariates. Am J Epidemiol1980;112:564–9.158. Caroll RJ. Covariance analysis in generalizedlinear measurement error models. Stat Med1989;8:1075–93.159. Savitz DA, Baron AE. Estimating and correctingfor confounder misclassification. Am J Epidemiol1989;129:1062–71.160. Knuiman MW, Divitini ML, Buzas JS, FitzgeraldPEB. Adjustment for regression dilution inepidemiological regression analyses. Ann Epidemiol1998;8:56–63.161. Rosner B, Spiegelman D, Willet WC. Correction oflogistic regression relative risk estimates andconfidence intervals for measurement error: thecase of multiple covariates measured with error.Am J Epidemiol 1990;132:734–45.162. Greenland S, Robins JM. Confounding andmisclassification. Am J Epidemiol 1985;122:495–506.163. Prentice RL. Design issues in cohort studies. StatMethods Med Res 1995;4:273–92.164. Gail MH, Wieand S, Piantadosi S. Biased estimatesof treatment effect in randomized experimentswith nonlinear regressions and omitted covariates.Biometrika 1984;71:431–44.165. Ford I, Norrie J. The role of covariates inestimating treatment effects and risk in long-termclinical trials. Stat Med 2002;21:2899–908.166. Hosmer DW, Lemeshow S. Applied logisticregression. New York: Wiley; 1989.167. Bancroft TA. Analysis and inference forincompletely specified models involving the use ofpreliminary tests of significance. Biometrics1964;20:247–439.168. Mickey RM, Greenland S. The impact ofconfounder selection criteria on effect estimation.Am J Epidemiol 1989;129:125–37.169. Maldonado G, Greenland S. Simulation study ofconfounder-selection strategies. Am J Epidemiol1993;138:923–36.170. Drake C. Effects of misspecification of thepropensity score on estimators of treatment effect.Biometrics 1993;49:1231–6.171. Charpentier PA, Bogardus ST, Inouye SK. Analgorithm for prospective individual matching in anon-randomized clinical trial. J Clin Epidemiol2001;54:1166–73.172. Johnston SC. Identifying confounding byindication through blinded prospective review. AmJ Epidemiol 2001;154:276–84.173. Phillips AN, Davey Smith G. The design ofprospective epidemiological studies: more subjectsor better measurements? J Clin Epidemiol 1993;46:1203–11.

Health Technology Assessment 2003; Vol. 7: No. 27174. Psaty BM, Koepsell TD, Lin D, Weiss NS,Siscovick DS, Rosendaal FR, et al. Assessment andcontrol for confounding by indication inobservational studies. J Am Geriatr Soc 1999;47:749–54.175. Khan KS, ter Riet, G, Glanville J, Sowden AJ,Kleijnen J. Undertaking systematic reviews ofresearch on effectiveness. CRD Report No. 4.Update. York: University of York; 2001.176. Audet N, Gagnon R, Ladouceur R, Marcil M. Howeffective is the teaching of critical analysis ofscientific publications? Review of studies and theirmethodological quality. CMAJ 1993;148:945–52.177. Campos-Outcalt D, Senf J, Watkins AJ, Bastacky S.The effects of medical school curricula, faculty rolemodels, and biomedical research support onchoice of generalist physician careers: a review andquality assessment of the literature. Acad Med1995;70:611–19.178. Carter J, Verhoef MJ. Efficacy of self-help andalternative treatments of premenstrual syndrome.Womens Health Issues 1994;4:130–7.179. Cochrane Musculoskeletal Injuries Group.Assessment of methodological quality of includedtrials. In The Cochrane Library, Issue 4. Oxford:Update Software; 2002.180. Cuddy PG, Elenbaas RM, Elenbaas JK. Evaluatingthe medical literature. Part I: abstract, introduction,methods. Ann Emerg Med 1983;12:549–55.181. Gardner MJ, Machin D, Campbell MJ. Use ofcheck lists in assessing the statistical content ofmedical studies. BMJ 1986;292:810–2.182. Glantz JC, McNanley TJ. Active management oflabor: a meta-analysis of cesarean delivery rates fordystocia in nulliparas. Obstet Gynecol Surv 1997;52:497–505.183. Greenhalgh T. Assessing the methodologicalquality of published papers. BMJ 1997;315:305–8.184. Gurman A, Kniskern D. Research on marital andfamily therapy: progress, perspective and prospect.In Garfield S, Bergin A, editors. Handbook ofpsychotherapy and behavior change: an empiricalanalysis. New York: Wiley; 1978. pp. 817–901.185. Cole MG, Primeau F, McCusker J. Effectiveness ofinterventions to prevent delirium in hospitalizedpatients: A systematic review. CMAJ 1996;155:1263–8.186. Kay EJ, Locker D. Is dental health educationeffective? A systematic review of current evidence.Community Dent Oral Epidemiol 1996;24:231–5.187. Kreulen CM, Creugers NH, Meijering AC. Metaanalysisof anterior veneer restorations in clinicalstudies. J Dent 1998;26:345–53.188. Kwakkel G, Wagenaar RC, Koelman TW,Lankhorst GJ, Koetsier JC. Effects of intensity ofrehabilitation after stroke. A research synthesis.Stroke 1997;28:1550–6.189. Lee TM, Chan CC, Paterson JG, Janzen HL,Blashko CA. Spectral properties of phototherapyfor seasonal affective disorder: a meta-analysis.Acta Psychiatr Scand 1997;96:117–21.190. Levine J. Trial Assessment Procedure Scale (TAPS).Bethesda, MD: Department of Health and HumanServices, Public Health Service, Alcohol, DrugAbuse and Mental Health Administration,National Institute of Mental Health; 1980.191. MacMillan HL, MacMillan JH, Offord DR, GriffithDL, MacMillan A. Primary prevention of childphysical abuse and neglect: a critical review. Part 1.J Child Psychol Psychiatry 1994;35:835–56.192. Maziak DE, Meade MO, Todd TR. The timing oftracheotomy: a systematic review. Chest 1998;114:605–9.193. Meijman F, Melker de RA. The extent of interandintra-reviewer agreement on the classificationand assessment of designs of single-practiceresearch. Fam Pract 1995;12:93–7.194. Barley ZA. Assessment of quality of studies forinclusion in meta-analyses. Doctoral dissertation,University of Colorado; 1989. Dissertation AbstractsInternational 1989;50A:575.195. Moncrieff J, Drummond DC. The quality ofalcohol treatment research: an examination ofinfluential controlled trials and development of aquality rating system. Addiction 1998;93:811–23.196. Morley JA, Finney JW, Monahan SC, Floyd AS.Alcoholism treatment outcome studies,1980–1992: methodological characteristics andquality. Addict Behav 1996;21:429–43.197. Mulrow CD, Lichtenstein MJ. Blood glucose anddiabetic retinopathy: a critical appraisal of newevidence. J Gen Intern Med 1986;1:73–7.198. Salisbury C. What is the impact of different modelsof care on patients’ quality of life, psychologicalwell-being or motivation? Appropriate and costeffective models of service delivery in palliativecare: report March 1996–July 1997. Bristol:Division of Primary Health Care, University ofBristol; 1997.199. Schechter MT, Leblanc FE, Lawrence VA. Criticalappraisal of published research. In Mulder D,McPeek B, Troidl H, Spitzer W, McKneally M,Weschler A, editors. Principles and practice ofresearch: strategy for surgical investigators. 2nded. New York: Springer; 1991. pp. 81–7.200. Sheldon TA, Song F, Davey Smith G. Criticalappraisal of the medical literature: how to assesswhether health-care interventions do more harmthan good. In Drummond MF, Maynard A,Wells N, editors. Purchasing and providing cost101© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

References102effective health care. London: ChurchillLivingstone; 1993. pp. 31–48.201. Vickers AJ. Can acupuncture have specific effectson health? A systematic review of acupunctureantiemesis trials. J R Soc Med 1996;89:303–11.202. Wingood GM, DiClemente RJ. HIV sexual riskreduction interventions for women: a review. Am JPrev Med 1996;12:209–17.203. Wright J, Dye R. Systematic review on obstructivesleep apnoea: its effect on health and benefit oftreatment. Leeds: University of Leeds; 1995.pp. 1–60.204. Hayes C, Antzcak-Bouckoms A, Burdick E. Qualityassessment and meta-analysis of systemictetracycline use in chronic adult periodontitis.J Clin Periodontol 1992;19:164–8.205. Norman GR, Shannon SI. Effectiveness ofinstruction in critical appraisal (evidence-basedmedicine) skills: a critical appraisal. CMAJ 1998;158:177–81.206. Avis M. Reading research critically. II. Anintroduction to appraisal: assessing the evidence.J Clin Nurs 1994;3:271–7.207. Bailar JC, Louis TA, Lavori PW, Polansky M.Studies without internal controls. N Engl J Med1984;311:156–62.208. Bauman LJ, Drotar D, Leventhal JM, Perrin EC,Pless IB. A review of psychosocial interventions forchildren with chronic health conditions. Pediatrics1997;100:244–51.209. Berard A, Bravo G, Gauthier P. Meta-analysis ofthe effectiveness of physical activity for theprevention of bone loss in postmenopausalwomen. Osteoporos Int 1997;7:331–7.210. Beyer WE, Palache AM, Osterhaus AD.Comparison of serology and reactogenicitybetween influenza subunit vaccines and wholevirus or split vaccines. A review and meta-analysisof the literature. Clin Drug Invest 1998;15:1–12.211. Bland JM, Jones DR, Bennett S, Cook DG, HainesAP, MacFarlane AJ. Is the clinical trial evidenceabout new drugs statistically adequate? Br J ClinPharmacol 1985;19:155–60.212. Boers M, Ramsden M. Longacting drugcombinations in rheumatoid arthritis: a formaloverview. J Rheumatol 1991;18:316–24.213. Borghouts JAJ, Koes BW, Bouter LM. The clinicalcourse and prognostic factors of non-specific neckpain: a systematic review. Pain 1998;77:1–13.214. Brown SA. Measurement of quality of primarystudies for meta-analysis. Nurs Res 1991;40:352–5.215. Brown SA. Studies of educational interventionsand outcomes in diabetic adults: a meta-analysisrevisited. Patient Educ Couns 1990;16:189–215.216. Brown SA, Grimes DE. A meta-analysis of nursepractitioners and nurse midwives in primary care.Nurs Res 1995;44:332–9.217. Brown SA, Upchurch S, Anding R, Winter M,Ramirez G. Promoting weight loss in type IIdiabetes. Diabetes Care 1996;19:613–24.218. Callahan CM, Dittus RS, Katz BP. Oralcorticosteroid therapy for patients with stablechronic obstructive pulmonary disease: a metaanalysis.Ann Intern Med 1991;114:216–23.219. Canadian Task Force on the Periodic HealthExamination. The periodic health examination.CMAJ 1979;121:1193–254.220. Canadian Task Force on the Periodic HealthExamination. Methodology. In Canadian TaskForce on the Periodic Health Examination. TheCanadian Guide to Clinical Preventive HealthCare. Ottawa: Health Canada; 1994. p. xxxvii.221. Watt D, Verma S, Flynn L. Wellness programs: areview of the evidence. CMAJ 1998;158:224–30.222. Catre MG. Anticoagulation in spinal surgery. Acritical review of the literature. Can J Surg1997;40:413–19.223. Loblaw DA, Laperriere NJ. Emergency treatmentof malignant extradural spinal cord compression:an evidence-based guideline. J Clin Oncol 1998;16:1613–24.224. Richard CS, McLeod RS. Follow-up of patientsafter resection for colorectal cancer: a positionpaper of the Canadian Society of SurgicalOncology and the Canadian Society of Colon andRectal Surgeons. Can J Surg 1997;40:90–100.225. Griffiths AM, Sherman PM. Colonoscopicsurveillance for cancer in ulcerative colitis: acritical review. J Pediatr Gastroenterol Nutr 1997;24:202–10.226. Cao WC, Van der Ploeg CP, Plaisier AP, van derSluijs IJ, Habbema JD. Ivermectin for thechemotherapy of bancroftian filariasis: a metaanalysisof the effect of single treatment. Trop MedInt Health 1997;2:393–403.227. Carlson M, Fanchiang SP, Zemke R, Clark F. Ameta-analysis of the effectiveness of occupationaltherapy for older persons. Am J Occup Ther 1996;50:89–98.228. Ball C, Sackett D, Phillips B, Haynes B, Straus S.Oxford Centre for Evidence-based MedicineLevels of Evidence (May 2001).http://minerva.minervation.com/cebm/docs/levels.html#levels. 2002.229. Gam AN, Johannsen F. Ultrasound therapy inmusculoskeletal disorders: a meta-analysis. Pain1995;63:85–91.230. Cheatham ML, Chapman WC, Key SP, Sawyers JL.A meta-analysis of selective versus routine

Health Technology Assessment 2003; Vol. 7: No. 27nasogastric decompression after electivelaparotomy. Ann Surg 1995;221:469–76.231. Ciliska D, Hayward S, Thomas H, Mitchell A,Dobbins M, Underwood J, et al. A systematicoverview of the effectiveness of home visiting as adelivery strategy for public health nursinginterventions. Can J Public Health 1996;87:193–8.232. Ploeg J, Ciliska D, Dobbins M, Hayward S,Thomas H, Underwood J. A systematic overview ofadolescent suicide prevention programs. Can JPublic Health 1996;87:319–24.233. Cochrane Effective Practice and Organisation ofCare Group. Assessment of methodological quality.The Cochrane Library, Issue 4. Oxford: UpdateSoftware; 2002.234. Oakley A, Fullerton D, Holland J. Behaviouralinterventions for HIV/AIDS prevention. AIDS1995;9:479–86.235. Towler B, Irwig L, Glasziou P, Kewenter J,Weller D, Silagy C. A systematic review of theeffects of screening for colorectal cancer using thefaecal occult blood test, Hemoccult. BMJ 1998;317:559–65.236. Cohen JE, Goel V, Frank JW, Bombardier C, PelosoP, Guillemin F. Group education interventions forpeople with low back pain: an overview of theliterature. Spine 1994;19:1214–22.237. Colditz GA, Brewer TF, Berkey CS, Wilson ME,Burdick E, Fineberg HV, et al. Efficacy of BCGvaccine in the prevention of tuberculosis. Metaanalysisof the published literature. JAMA 1994;271:698–702.238. Cook DJ, Guyatt GH, Laupacis A, Sackett DL.Rules of evidence and clinical recommendationson the use of antithrombotic agents. Chest 1992;102:305s–11s.239. Schmidt-Nowara W, Lowe A, Wiegland L,Cartwright R, Perez-Guerra F, Menn S. Oralappliances for the treatment of snoring andobstructive sleep apnoea. Sleep 1995;18:501–10.240. Chaulk CP, Kazandjian VA. Directly observedtherapy for treatment completion of pulmonarytuberculosis: Consensus Statement of the PublicHealth Tuberculosis Guidelines Panel. JAMA1998;279:943–8.241. Mayo-Smith MF. Pharmacological management ofalcohol withdrawal. A meta-analysis and evidencebasedpractice guideline. American Society ofAddiction Medicine Working Group onPharmacological Management of AlcoholWithdrawal. JAMA 1997;278:144–51.242. Cox GL, Merkel WT. A qualitative review ofpsychosocial treatments for bulimia. J Nerv MentDis 1989;177:77–84.243. Cuijpers P. Psychological outreach programmes forthe depressed elderly: a meta-analysis of effectsand dropout. Int J Geriatr Psychiatry 1998;13:41–8.244. de Craen AJ, Di Giulio G, Lampe SchoenmaeckersJE, Kessels AG, Kleijnen J. Analgesic efficacy andsafety of paracetamol–codeine combinations versusparacetamol alone: a systematic review. BMJ 1996;313:321–5.245. de Kruif YP, van Wegen EE. Pelvic floor muscleexercise therapy with myofeedback for women withstress urinary incontinence: a meta-analysis.Physiotherapy 1996;82:107–13.246. de Oliveira IR, de Sena EP, Pereira EL, MirandaAM, de Oliveira NF, Ribeiro MG, et al.Haloperidol blood levels and clinical outcome: ameta-analysis of studies relevant to testing thetherapeutic window hypothesis. J Clin Pharm Ther1996;21:229–36.247. DerSimonian R, Charette LJ, McPeek B, MostellerF. Reporting on methods in clinical trials. N Engl JMed 1982;306:1332–7.248. Curatolo M, Petersen-Felix S, Scaramozzino P,Zbinden AM. Epidural fentanyl, adrenaline andclonidine as adjuvants to local anaesthetics forsurgical analgesia: meta-analyses of analgesia andside-effects. Acta Anaesthesiol Scand 1998;42:910–20.249. Aronson R, Offman HJ, Joffe RT, Naylor CD.Triiodothyronine augmentation in the treatmentof refractory depression. A meta-analysis. Arch GenPsychiatry 1996;53:842–8.250. Devine EC. Effects of psychoeducational care foradult surgical patients: a meta-analysis of 191studies. Patient Educ Couns 1992;19:127–42.251. Devine EC, Cook TD. A meta-analytic analysis ofeffects of psycho-educational interventions onlength of post-surgical hospital stay. Nurs Res1983;32:267–74.252. Devine EC, Reifschneider E. A meta-analysis of theeffects of psychoeducational care in adults withhypertension. Nurs Res 1995;44:237–45.253. Devine EC, Westlake SK. The effects ofpsychoeducational care provided to adults withcancer: meta-analysis of 116 studies. Oncol NursForum 1995;22:1369–81.254. Brown SA. Effects of educational interventions indiabetes care: a meta-analysis of findings. Nurs Res1988;37:223–30.255. Durlak JA, Wells AM. Primary prevention mentalhealth programs: the future is exciting. Am JCommunity Psychol 1997;25:233–43.256. Elvik R. The safety value of guardrails and crashcushions: a meta-analysis of evidence fromevaluation studies. Accid Anal Prev 1995;27:523–49.257. Esdaile JM, Horwitz RI. Observational studies ofcause-effect relationships: an analysis ofmethodological problems as illustrated by the103© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

References104conflicting data for the role of oral contraceptivesin the etiology of rheumatoid arthritis. J Chron Dis1986;39:841–52.258. Farquhar WW, Krumboltz JD. A checklist forevaluating experimental research in psychologyand education. J Educ Res 1959;52:353–4.259. Fox JH. Criteria of good research. Phi DeltaKappan 1958;39:284–6.260. Collingsworth S, Gould D, Wainwright SP. Patientself-administration of medication: a review of theliterature. Int J Nurs Stud 1997;34:256–69.261. Gansevoort RT, Sluiter WJ, Hemmelder MH,de Zeeuw D, de Jong PE. Antiproteinuric effect ofblood-pressure-lowering agents: a meta-analysis ofcomparative trials. Nephrol Dial Transplant 1995;10:1963–74.262. Garber BG, Hebert PC, Yelle JD, Hodder RV,McGowan J. Adult respiratory distress syndrome:a systematic overview of incidence and risk factors.Crit Care Med 1996;24:687–95.263. Garg A, Shiau J, Guyatt G. The ineffectiveness ofimmunosuppressive therapy in lymphocyticmyocarditis: an overview. Ann Intern Med 1998;128:317–22.264. Callahan CM, Drake BG, Heck DA, Dittus RS.Patient outcomes following unicompartmental orbicompartmental knee arthroplasty: a metaanalysis.J Arthroplasty 1995;10:141–50.265. Gifford RH, Feinstein AR. A critique ofmethodology in studies of anticoagulant therapyfor acute myocardial infarction. N Engl J Med1969;280:351–7.266. Good M. Effects of relaxation and music onpostoperative pain: a review. J Adv Nurs 1996;24:905–14.267. Fiscella K. Does prenatal care improve birthoutcomes? A critical review. Obstet Gynecol 1995;85:468–79.268. Gray MA, McPherson IG. Agoraphobia: a criticalreview of methodology in behavioural treatmentresearch. Curr Psychol Rev 1982;2:19–45.269. Gyorkos TW, Tannenbaum TN, Abrahamowicz M,Bedard L, Carsley J, Franco ED, et al. Evaluationof the effectiveness of immunization deliverymethods. Can J Public Health 1994;85:S14–30.270. Riben PD, Mathias RG, Campbell E, Wiens M.The evaluation of the effectiveness of routinerestaurant inspections and education of foodhandlers: critical appraisal of the literature. Can JPublic Health 1994;85:S56–60.271. Oxman AD, Scott EA, Sellors JW, Clarke JH,Millson ME, Rasooly I, et al. Partner notificationfor sexually transmitted diseases: an overview ofthe evidence. Can J Public Health 1994;85:S41–7.272. Hancock L, Sanson Fisher RW, Redman S, BurtonR, Burton L, Butler J, et al. Community action forhealth promotion: a review of methods andoutcomes 1990–1995. Am J Prev Med 1997;13:229–39.273. Haynes RB, Sackett DL, Taylor DW, Hackett BC,Luterbach E, Cloak JR. An annotated bibliography(including notes on methodologic standards forcompliance research). In Sackett DL, Haynes RB,editors. Compliance with therapeutic regimens.Baltimore, MD: Johns Hopkins University Press;1976. pp. 193–7.274. Mullen PD, Mains DA, Velez R. A meta-analysis ofcontrolled trials of cardiac patient education.Patient Educ Couns 1992;19:143–56.275. Mullen PD, Green LW, Persinger G. Clinical trialsof patient education for chronic conditions: acomparative meta-analysis of intervention types.Prev Med 1985;14:753–81.276. Heacock H, Koehoorn M, Tan J. Applyingepidemiological principles to ergonomics: achecklist for incorporating sound design andinterpretation of studies. Appl Ergon1997;28:165–72.277. Hedges LV. Issues in meta-analysis. Review ofResearch in Education 1986;13:353–98.278. Helfenstein U, Steiner M. Fluoride varnishes(Duraphat): a meta-analysis. Comm Dent OralEpidemiol 1994;22:1–5.279. Heneghan AM, Horwitz SM, Leventhal JM.Evaluating intensive family preservation programs:a methodological review. Pediatrics 1996;97:535–42.280. Hill MJ, Blane HT. Evaluation of psychotherapywith alcoholics: a critical review. Q J Stud Alcohol1967;28:76–104.281. Hines DC, Goldziehr JW. Clinical investigation: aguide to its evaluation. Am J Obstet Gynecol 1969;105:450–87.282. Hoag M, Burlingame G. Evaluating theeffectiveness of child and adolescent grouptreatment: a meta-analytic review. J Clin ChildPsychol 1997;26:234–46.283. Hoffman RM, Wheeler KJ, Deyo RA. Surgery forherniated lumbar discs: a literature synthesis.J Gen Intern Med 1993;8:487–96.284. Howell WH, McNamara DJ, Tosca MA, Smith BT,Gaines JA. Plasma lipid and lipoprotein responsesto dietary fat and cholesterol: a meta-analysis. Am JClin Nutr 1997;65:1747–64.285. Jabbour M, Osmond MH, Klassen TP. Life supportcourses: are they effective? Ann Emerg Med 1996;28:690–8.286. Lucassen PL, Assendelft WJ, Gubbels JW, van EijkJT, van Geldrop WJ, Neven AK. Effectiveness of

Health Technology Assessment 2003; Vol. 7: No. 27treatments for infantile colic: systematic review.BMJ 1998;316:1563–9.287. Ernst E, Pittler MH. The effectiveness ofacupuncture in treating acute dental pain: asystematic review. Br Dent J 1998;184:443–7.288. Saint S, Elmore JG, Sullivan SD, Emerson SS,Koepsell TD. The efficacy of silver alloy-coatedurinary catheters in preventing urinary tractinfection: a meta-analysis. Am J Med 1998;105:236–41.289. Johnston ME, Langton KB, Haynes RB, MathieuA. Effects of computer-based clinical decisionsupport systems on clinician performance andpatient outcome. A critical appraisal of research.Ann Intern Med 1994;120:135–42.290. Sullivan F, Mitchell E. Has general practitionercomputing made a difference to patient care? Asystematic review of published reports. BMJ1995;311:848–52.291. Kasiske BL, Ma JZ, Kalil RS, Louis TA. Effects ofantihypertensive therapy on serum lipids. AnnIntern Med 1995;122:133–41.292. Kasiske BL, Lakatua JD, Ma JZ, Louis TA. A metaanalysisof the effects of dietary protein restrictionon the rate of decline in renal function. Am JKidney Dis 1998;31:954–61.293. Kasiske BL, Heimduthoy K, Ma JZ. Electivecyclosporine withdrawal after renal transplantation– a meta-analysis. JAMA 1993;269:395–400.294. King PR. Methodological guidelines for readingdrug evaluation research. Can J Psychiatry1984;29:575–82.295. King PR. Appraising the quality of drugevaluationresearch: I. A method of meta-analysisfor acute treatment medications. Can J Psychiatry1990;35:316–9.296. King PR. Appraising the quality of drugevaluationresearch: II. A methodological reviewprocedure for prophylactic treatment trials. J ClinPsychopharmacol 1990;10:22–6.297. Kingery WS. A critical review of controlled clinicaltrials for peripheral neuropathic pain andcomplex regional pain syndromes. Pain1997;73:123–39.298. Kinney MR, Burfitt SN, Stullenbarger E, Rees B,DeBolt MR. Quality of life in cardiac patientresearch: a meta analysis. Nurs Res1996;45:173–80.299. Klassen T, Verhey FR, Sneijders GH, Rozendaal N,De Vet HC, Van Praag HM. Treatment ofdepression in Parkinson’s disease: a meta-analysis.J Neuropsychiatry Clin Neurosci 1995;7:281–6.300. Kleijnen J, Knipschild P. Mistletoe treatment forcancer. Review of controlled trials in humans.Phytomedicine 1994;1:255–60.301. Kleijnen J, Knipschild P. Hyperbaric oxygen formultiple sclerosis. Review of controlled trials. ActaNeurol Scand 1995;91:330–4.302. Ernst E, Barnes J. Are homeopathic remedieseffective for delayed-onset muscle soreness? Asystematic review of placebo-controlled trials.Perfusion 1998;11:4–8.303. Lijesen GKS, Theeuwen I, Assendelft WJJ, Van derWal G. The effect of human chorionicgonadotropin (hCG) in the treatment of obesity bymeans of the simeons therapy: a criteria-basedmeta-analysis. Br J Clin Pharmacol 1995;40:237–43.304. Verhagen AP, de Vet HC, de Bie RA, Kessels AG,Boers M, Knipschild PG. Taking baths: the efficacyof balneotherapy in patients with arthritis. Asystematic review. J Rheumatol 1997;24:1964–71.305. Kristensen TS. Cardiovascular diseases and thework environment. A critical review of theepidemiologic literature on nonchemical factors.Scand J Work Environ Health 1989;15:165–79.306. Krause N, Dasinger LK, Neuhauser F. Modifiedwork and return to work: a review of the literature.J Occup Rehab 1998;8:113–39.307. Krogh CL. A checklist system for critical review ofmedical literature. Med Educ 1985;19:392–5.308. Kulik CL, Kulik JA, Bangert-Drowns RL.Effectiveness of mastery learning programs: ameta-analysis. Rev Educ Res 1990;60:265–99.309. Detsky AS, Baker JP, O’Rourke K, Goel V.Perioperative parenteral nutrition: a meta-analysis.Ann Intern Med 1987;107:195–203.310. Labrecque M. Evaluation critique des publicationsmédicales: mise à jour pratique. Can Fam Physician1989;35:792–5.311. Leboeuf C. A review of data reports published inthe Journal of Manipulative and PhysiologicalTherapeutics from 1986 to 1988. J ManipulativePhysiol Ther 1990;13:89–95.312. Lee C-M, Picard M, Blain M-D. A methodologicaland substantive review of intervention outcomestudies for families undergoing divorce. J FamPsychol 1994;8:3–15.313. Cooper BC, Mullins PR, Jones MR, Lang SD.Clinical efficacy of roxithromycin in the treatmentof adults with upper and lower respiratory tractinfection due to Haemophilus influenzae. A metaanalysisof 12 clinical studies. Drug Invest 1994;7:299–314.314. Linde K, Melchart D. Randomized controlledtrials of individualized homeopathy: a state-ofthe-artreview. J Altern Complement Med 1998;4:371–88.315. Lionel ND, Herxheimer A. Assessing reports oftherapeutic trials. BMJ 1970;3:637–40.105© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

References106316. Littenberg B, Weinstein LP, McCarren M, Mead T,Swiontkowski MF, Rudicel SA, et al. Closedfractures of the tibial shaft. A meta-analysis ofthree methods of treatment. J Bone Joint Surg Am1998;80:174–83.317. Longnecker MP, Berlin JA, Orza MJ, ChalmersTC. A meta-analysis of alcohol consumption inrelation to risk of breast cancer. JAMA 1988;260:652–6.318. Luborsky L, Singer B, Luborsky L. Comparativestudies of psychotherapies. Is it time that“everyone has won and all must have prizes?” ArchGen Psychiatry 1975;32:995–1008.319. Lyons LC, Woods PJ. The efficacy of rationalemotivetherapy: a quantitative review of theoutcome research. Clin Psychol Rev 1991;11:357–69.320. Mahon WA, Daniel EE. A method for theassessment of reports of drug trials. CMAJ 1964;90:565–9.321. Margetts BM, Thompson RL, Key T, Duffy S,Nelson M, Bingham S, et al. Development of ascoring system to judge the scientific quality ofinformation from case-control and cohort studiesof nutrition and disease. Nutr Cancer 1995;24:231–9.322. Margolis LH. A critical review of studies ofnewborn discharge timing. Clin Pediatr Phila 1995;34:626–34.323. Marrs RW. A meta-analysis of bibliotherapystudies. Am J Community Psychol 1995;23:843–70.324. Massy ZA, Ma JZ, Louis TA, Kasiske BL. Lipidloweringtherapy in patients with renal disease.Kidney Int 1995;48:188–98.325. McAweeney MJ, Tate DG, McAweeney W.Psychosocial interventions in the rehabilitation ofpeople with spinal cord injury: a comprehensivemethodological inquiry. SCI Psychosocial Process1997;10:58–66.326. Flint AJ. Effects of respite care on patients withdementia and their caregivers. Int Psychogeriatr1995;7:505–17.327. Cole M-G. The impact of geriatric medical serviceson mental state. Int Psychogeriatr 1993;5:91–101.328. McCusker J, Cole M, Keller E, Bellavance F,Berard A. Effectiveness of treatments of depressionin older ambulatory patients. Arch Intern Med1998;158:705–12.329. Meinert CL. Essential design features of acontrolled clinical trial. In Clinical trials: design,conduct and analysis. Oxford: Oxford UniversityPress; 1986. pp. 65–70.330. Morin CM, Culbert JP, Schwartz SM.Nonpharmacological interventions for insomnia: ameta-analysis of treatment efficacy. Am J Psychiatry1994;151:1172–80.331. Morris SB, Turner CW, Szykula SA. Psychotherapyoutcome research: an application of a new methodfor evaluating research methodology. Psychotherapy1988;25:18–26.332. Murtagh DR, Greenwood KM. Identifying effectivepsychological treatments for insomnia: a metaanalysis.J Consult Clin Psychol 1995;63:79–89.333. Naldi L, Parazzini F, Cainelli T. Role of topicalimmunotherapy in the treatment of alopeciaareata. Quality analysis of articles publishedbetween January 1977 and January 1988 aboutthree treatments. Reading Group. J Am AcadDermatol 1990;22:654–6.334. Naylor CD, Detsky AS, O’ Rourke K, Fonberg E.Does treatment with essential amino acids andhypertonic glucose improve survival in acute renalfailure? A meta-analysis. Ren Fail 1987;10:141–52.335. Camma C, Almasio P, Craxi A. Interferon astreatment for acute hepatitis C. A meta-analysis.Dig Dis Sci 1996;41:1248–55.336. Nielsen ME, Reilly PL. A guide to understandingand evaluating research articles. Gifted Child Q1985;29:90–2.337. Nyberg G. Assessment of papers of clinical trials.Med J Aust 1974;2:381.338. EPI Centre. Review of effectiveness of sexualhealth promotion interventions for young people.London: Social Science Research Unit, Universityof London Institute of Education; 1996.339. Oakley A, Fullerton D. Sexual health educationinterventions for young people: a methodologicalreview. BMJ 1995;310:158–62.340. Foxcroft DR, Lister-Sharp D, Lowe G. Alcoholmisuse prevention for young people: a systematicreview reveals methodological concerns and lack ofreliable evidence of effectiveness. Addiction 1997;92:531–7.341. Ogilvie-Harris DJ, Gilbart M. Treatmentmodalities for soft tissue injuries of the ankle: acritical review. Clin J Sport Med 1995;5:175–86.342. Osberg TM. Teaching psychotherapy outcomeresearch methodology using a research-basedchecklist. Teach Psychol 1997;24:271–4.343. Oxman A. No magic bullets: a systematic review of102 trials of interventions to help health careprofessionals deliver services more effectively orefficiently. London: North East Thames RegionalHealth Authority; 1994.344. Powe NR, Tielsch JM, Schein OD, Luthra R,Steinberg EP. Rigor of research methods in studiesof the effectiveness and safety of cataractextraction with intraocular lens implantation.

Health Technology Assessment 2003; Vol. 7: No. 27Cataract Patient Outcome Research Team. ArchOphthalmol 1994;112:228–38.345. Puig Barbera J, Marquez Calderon S. Effectivenessof influenza vaccine in the elderly. A critical reviewof the bibliography. Med Clin (Barc) 1995;105:645–8.346. Rey JM, Walter G. Half a century of ECT use inyoung people. Am J Psychiatry 1997;154:595–602.347. Ried LD, Carter KA, Ellsworth A. Acetazolamideor dexamethasone for prevention of acutemountain sickness: a meta-analysis. J WildernessMed 1994;5:34–48.348. Rowe DE, Bernstein SM, Riddick MF, Adler F,Emans JB, Gardner Bonneau D. A meta-analysis ofthe efficacy of non-operative treatments foridiopathic scoliosis. J Bone Joint Surg Am 1997;79:664–74.349. Sackett DL, Haynes RB, Tugwell P. Clinicalepidemiology: a basic science for clinicalmedicine. Boston, MA: Little Brown;1985.350. Heyland DK, Cook DJ, Guyatt GH. Enteralnutrition in the critically ill patient: a criticalreview of the evidence. Intensive Care Med 1993;19:435–42.351. Ciliska D, Kelly C, Petrov N, Chalmers J. A reviewof weight loss interventions for obese people withnon-insulin-dependent diabetes mellitus. Can JDiabetes Care 1995;19:10–15.352. Choi HW, Breman JG, Teutsch SM, Liu S,Hightower AW, Sexton JD. The effectiveness ofinsecticide-impregnated bed nets in reducingcases of malaria infection: a meta-analysis ofpublished results. Am J Trop Med Hyg1995;52:377–82.353. Sellers DE, Crawford SL, Bullock K, McKinlay JB.Understanding the variability in the effectivenessof community heart health programs: a metaanalysis.Soc Sci Med 1997;44:1325–39.354. Selley S, Donovan J, Faulkner A, Coast J, Gillatt D.Diagnosis, management and screening of earlylocalised prostate cancer. Health Technol Assess1997;1(2).355. Shaver JP, Curtis CK, Jesunathadas J, Strong CJ.The modification of attitudes towards persons withdisabilities: is there a best way? Int J Special Educ1989;4:33–57.356. Simons MP, Kleijnen J, van Geldere D, HoitsmaHF, Obertop H. Role of the Shouldice techniquein inguinal hernia repair: a systematic review ofcontrolled trials and a meta-analysis. Br J Surg1996;83:734–8.357. Slavin RE. Best-evidence synthesis: an alternativeto meta-analytic and traditional reviews. EducResearcher 1986;15:5–11.358. Schoemaker C. Does early intervention improvethe prognosis in anorexia nervosa? A systematicreview of the treatment-outcome literature. Int JEat Disord 1997;21:1–15.359. Smeenk FW, van Haastregt JC, de Witte LP,Crebolder HM. Effectiveness of home careprogrammes for patients with incurable canceron their quality of life and time spent inhospital: systematic review. BMJ1998;316:1939–44.360. Smith MC, Stullenbarger E. A prototype forintegrative review and meta-analysis of nursingresearch. J Adv Nurs 1991;16:1271–83.361. Smith NL, Glass GV. Meta-analysis ofpsychotherapy outcome studies. Am Psychol 1977;32:752–60.362. Burckhardt CS. The effect of therapy on themental health of the elderly. Res Nurs Health 1987;10:277–85.363. Krywanio ML. Meta-analysis of physiologicaloutcomes of hospital-based infant interventionprograms. Nurs Res 1994;43:133–7.364. Smith NL, Glass GV, Miller TI. Benefits ofpsychotherapy. Baltimore, MD: Johns HopkinsUniversity Press; 1980.365. Solomon MJ, McLeod RS. Clinical studies insurgical journals – have we improved? Dis ColonRectum 1993;36:43–8.366. Solomon DH, Bates DW, Panush RS, Katz JN.Costs, outcomes, and patient satisfaction byprovider type for patients with rheumatic andmusculoskeletal conditions: a critical review of theliterature and proposed methodologic standards.Ann Intern Med 1997;127:52–60.367. Stevens A, Raftery J. The stimulus for needsassessment: reforming health services. InStevens A, Raftery J, editors. Health care needsassessment. Oxford: Radcliffe Medical Press; 1994.pp. 11–30.368. Dodhia H, Miller E. Review of the evidence for theuse of erythromycin in the management ofpersons exposed to pertussis. Epidemiol Infect1998;120:143–9.369. Stock S. Workplace ergonomic factors and thedevelopment of musculoskeletal disorders of theneck and upper limbs: a meta-analysis. Am J IndMed 1991;19:87–107.370. Suydam MN. An instrument for evaluatingexperimental educational research reports. J EducRes 1968;61:200–3.371. Floyd M, Scogin F. Effects of memory training onthe subjective memory functioning and mentalhealth of older adults: a meta-analysis. PsycholAging 1997;12:150–61.107© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

References108372. Tangkanakul C, Counsell CE, Warlow CP.Local versus general anaesthesia in carotidendarterectomy: a systematic review ofthe evidence. Eur J Vasc Endovasc Surg1997;13:491–9.373. Theis JG, Selby P, Ikizler Y, Koren G. Currentmanagement of the neonatal abstinencesyndrome: a critical analysis of the evidence. BiolNeonate 1997;71:345–56.374. Thomas JP, Lawrence TS. Common deficiencies ofNIDRR research applications. Am J Phys MedRehabil 1991;70:161–4.375. Thomas J, DeHueck A, Kleiner M, Newton J,Crowe J, Mahler S. To vibrate or not to vibrate:usefulness of the mechanical vibrator forclearing bronchial secretions. Physiother Can1995;47:120–5.376. Tuckman BW. A proposal for improving thequality of published research. Educ Researcher1990;19:2–25.377. US Preventive Services Task Force. Task forceratings. In US Preventive Services Task Force,editor. Guide to clinical preventive services.Baltimore, MD: Williams & Wilkins; 1996.pp. 861–85.378. US Preventive Services Task Force. Task forceratings. In US Preventive Services Task Force,editor. Guide to Clinical Preventive Services.Baltimore, MD: Williams & Wilkins; 1989.pp. 387–98.379. Robert G, Stevens A. Should general practitionersrefer patients directly to physical therapists? Br JGen Pract 1997;47:314–18.380. Long T, Soderstrom E. A critical appraisal ofpositioning infants in the neonatal intensive careunit. Phys Occup Ther Pediatr 1995;15:17–31.381. Witlin AG, Sibai BM. Magnesium sulfate therapyin preeclampsia and eclampsia. Obstet Gynecol1998;92:883–9.382. Grullon KE, Grimes DA. The safety of earlypostpartum discharge: a review and critique. ObstetGynecol 1997;90:860–5.383. Grimes DA. Medical abortion in early pregnancy:a review of the evidence. Obstet Gynecol 1997;89:790–6.384. van Balkom AJ, Bakker A, Spinhoven P, BlaauwBM, Smeenk S, Ruesink B. A meta-analysis of thetreatment of panic disorder with or withoutagoraphobia: a comparison ofpsychopharmacological, cognitive-behavioral, andcombination treatments. J Nerv Ment Dis 1997;185:510–16.385. van Dalen DB. A research checklist in education.Educ Admin Supervision 1958;44:174–81.386. Vickers AJ. Hypnotherapy for irritable bowelsyndrome. A report commissioned by North EastThames Regional Health Authority. London:Research Council for Complementary Medicine;1994.387. Ward M, Fetler M. What guidelines should befollowed in critically evaluating research reports.Nurs Res 1979;28:120–5.388. Weiler JM. Medical modifiers of sports injuries.The use of non-steroidal anti-inflammatory drugsin sports soft tissue injury. Clinc Sports Med 1992;11:625–44.389. Wolf RM. Judging educational research based onexperiments and surveys. Fundamentals ofeducational planning series No. 45. Paris: UnitedNations Educational, Scientific and CulturalOrganization; 1993.390. Woolf SH, Battista RN, Andeson GM, Logan AG,Wang E, Canadian Task Force on the PeriodicHealth Examination. Assessing the clinicaleffectiveness of preventive manoeuvres: analyticprinciples and systematic methods in reviewingevidence and developing clinical practicerecommendations. J Clin Epidemiol 1990;43:891–905.391. NHS Centre for Reviews and Dissemination. Reviewof the research on the effectiveness of health serviceinterventions to reduce variations in health. CRDReport 3. York: University of York, 1995. pp. 1–157.392. Zola P, Volpe T, Castelli G, Sismondi P, NicolucciA, Parazzini F, et al. Is the published literature areliable guide for deciding between alternativetreatments for patients with early cervical cancer?Int J Radiat Oncol Biol Phys 1989;16:785–97.393. ACCP/AACVPR Pulmonary RehabilitationGuidelines Panel. Pulmonary rehabilitation: jointACCP/AACVPR evidence-based guidelines. Chest1997;112:1363–96.394. Eastwood AE, Sheldon TA. Organisation of asthmacare: what differences does it make? A systematicreview of the literature. Qual Health Care 1996;5:134–43.395. Hearn J, Higginson IJ. Do specialist palliative careteams improve outcomes for cancer patients?Palliat Med 1998;12:317–32.396. MacMillan HL, MacMillan JH, Offord DR, GriffithDL, MacMillan A. Primary prevention of childsexual abuse and neglect: a critical review. Part 2. JChild Psychol Psychiatry 1994;35:857–76.397. Mathews F. Antioxidant nutrients in pregnancy: asystematic review of the literature. Nutr Res Rev1996;9:175–95.398. Parker J, Bisits A, Proietto AM. A systematic reviewof single-dose intramuscular methotrexate for thetreatment of ectopic pregnancy. Aust NZ J ObstetGynaecol 1998;38:145–50.

Health Technology Assessment 2003; Vol. 7: No. 27399. Powe NR, Schein OD, Gieser SC, Tielsch JM,Luthra R, Javitt J, et al. Synthesis of theliterature on visual acuity and complicationsfollowing cataract extraction with intraocularlens implantation. Cataract Patient OutcomeResearch Team. Arch Ophthalmol 1994;112:239–52.400. Snowdon SK, Stewart-Brown SL. Preschool visionscreening: results of a systematic review. CRDReport 9. York: University of York, 1997.pp. 1–114.401. Talley NJ, Owen BK, Boyce P, Paterson K.Psychological treatment for irritable bowelsyndrome: a critique of controlled treatment trials.Am J Gastroenterol 1996;91:277–83.402. Villalobos T, Arango C, Kubilis P, Rathore M.Antibiotic prophylaxis after basilar skullfractures: a meta-analysis. Clin Infect Dis1998;27:364–9.403. Poynard T. Evaluation de la qualitemethodologique des essais therapeutiquesrandomises. Presse Med 1988;17:315–18.404. Fiske DW, Hunt HF, Luborsky L, Orne MT et al.Planning of research on effectiveness ofpsychotherapy. American Psychologist1970;25:727–37.405. Bergin AE, Lambert MJ. The evaluation oftherapeutic outcomes. In Garfield SL, Bergin AE,editors. Handbook of psychotherapy and behaviorchange. New York: John Wiley; 1978. pp. 139–89.406. Jonsson C, Martens S, Sjoqvist F. [Attitude towardclinical tests of drugs – especially psychopharmaca].Nordisk Psykiatrisk Tidsskrift. 1969;23:281–9.407. Evans M, Pollock AV. A score system for evaluatingrandom control clinical trials of prophylaxis ofabdominal surgical wound infections. Br J Surg1985;72:256–60.109© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Health Technology Assessment 2003; Vol. 7: No. 27Appendix 1Search strategy for Chapters 3–5MEDLINE (1966–99) via Ovid(check adj list$).tw.check-list$.tw.checklist$.tw.1 or 2 or 3methodolog$.tw.(((check adj list$) or check-list$ or checklist$) adj5methodolog$).tw.quality.tw.validity.tw.validat$.tw.assess$.tw.evaluat$.tw.7 or 8 or 9 or 10 or 11study.tw.studies.tw.paper$.tw.article$.tw.literature.tw.report$.tw.research.tw.13 or 14 or 15 or 16 or 17 or 18 or 19(((check adj list$) or check-list$ or checklist$) adj5(quality or validity or validat$ or assess$ orevaluat$) adj5 (study or studies or paper$ orarticle$ or literature or report$ or research)).tw.bias$.tw.exp "bias (epidemiology)"/22 or 234 and 246 or 21 or 25((scale or scales) adj5 methodolog$).tw.((scale or scales) adj3 (quality or validity orvalidat$ or assess$ or evaluat$) adj3 (study orstudies or paper$ or article$ or literature or reportor reports or research)).tw.(inter adj rater adj reliability).tw.(interrater adj reliability).tw.(test adj retest adj reliability).tw.(testretest adj reliability).tw.(testre adj test adj reliability).tw.(test adj re adj test adj reliability).tw.(scales or scale).tw.29 or 30 or 31 or 32 or 33 or 344 and 36((scales or scale) adj3 ((inter adj rater adjreliability) or (interrater adj reliability) or (test adjretest adj reliability) or (testretest adj reliability) or(testre adj test adj reliability) or (test adj re adj testadj reliability))).tw.27 or 28 or 38evaluat$.ti.confounding.tw.(critical$ adj apprais$).tw.40 or 41 or 424 and 43case-control studies/(observational adj (study or studies or data)).tw.(non-random$ or nonrandom$).tw.(natural adj experiment$).tw.(quasi adj experiment$).tw.quasiexperiment$.tw.(non adj experiment$).tw.nonexperiment$.tw.intervention studies/cohort studies/45 or 46 or 47 or 48 or 49 or 50 or 51 or 52 or 53or 544 and 5546 or 47 or 48 or 49 or 50 or 51 or 527 or 8 or 9(((observational adj (study or studies or data)) or(non-random$ or nonrandom$) or (natural adjexperiment$) or (quasi adj experiment$) orquasiexperiment$ or (non adj experiment$) ornonexperiment$) adj5 (quality or validity orvalidat$)).tw.(assess$ or judg$ or measur$ or analy$ orevaluat$).ti.quality.ti.(critical$ adj apprais$).ti.(research or literature or report$ or paper$ orstudy or studies or article$).ti.((assess$ or judg$ or measur$ or analy$ orevaluat$) adj3 quality adj3 (research or literatureor report$ or paper$ or study or studies orarticle$)).ti.(critical$ adj apprais$ adj3 (research or literatureor report$ or paper$ or study or studies orarticle$)).ti.data interpretation, statistical/23 and 6655 and 6737 or 39 or 44 or 56 or 59 or 64 or 65 or 68((scales or scale) adj3 bias$).tw.(methodolog$ adj5 quality).ti.(bias$ adj3 ((observational adj (study or studies or111© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Appendix 1data)) or (non-random$ or nonrandom$) or(natural adj experiment$) or (quasi adjexperiment$) or quasiexperiment$ or (non adjexperiment$) or nonexperiment$)).tw.26 or 69 or 70 or 71 or 72(nonrandom$ adj allocation).tw.(non-random$ adj allocation).tw.(historical adj control$).tw.exp clinical trials/random allocation/research design/(gold adj standard).tw.(randomi#ed adj controlled adj trial$).tw.research design/55 or 77 or 78 or 7976 and 8380 and 8178 and 8274 or 75 or 84 or 85 or 86non random$.tw.nonrandom$.tw.allocat$.tw.((non random$ or nonrandom$) adj5 allocat$).tw.88 or 89 or 90 or 9173 or 87 or 92112

Health Technology Assessment 2003; Vol. 7: No. 27Appendix 2Data extraction forms113© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Appendix 2AuthorYearENDARESourcePublished as:Topic AreaTool purpose:Type of studiesQuality issuesType of toolIf Scale,Design1Quality1Design2Quality2Specify modified toolSeparate questions by study designdefines high quality thresholdweightingType/no of itemsTotalGenericTopic-specificWas choice of items justified?Quality items reported?How were items generated?Was scale validity assessed?Validity1Validity2Validity3Validity4Was scale reliability assessed?Reliability1Reliability2Reliability3Reliability4114Reliability5

Health Technology Assessment 2003; Vol. 7: No. 27Time to completeAverage time takenTested on sampleORDescribe methodDescribe studiesDescribe topic areaWhat items are included?Comments and/or useful referencesChecklist referencesEN noOther referencesEN NoHas been used by other reviews115© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Appendix 2Author:Accession No:Endnote No:Journal:Did the review use validity assessment?Did the review consider CMA?Year:Were results clearly reported by study design?Was the VA/CMA considered in the study synthesis?Type of review:Aim of intervention:Intervention type:Disease category:Literature searchDatabase1:Database2:Database3:Database4:Database5:Database6:Other search1:Other search2:Other search3:Other search4:Other search5:Other search6:Inclusion criteriaInclusion criteria1:Inclusion criteria2:No of studies considered:No of studies excluded:Languages included:Type/number of studiesTotal no of studies:No of RCTs:No of non-RCTs:Total no of patients:No patients:No patients:List type of non-RCTs:No of non-comp:No patients:List type of non-comp:Quality assessmentWas VA carried out?Was independent VA carried out?Same elig criteria?Similar study dates?Was same tool used for all designs?Type of tool:Tool methodSpecify tool:Study designs:Tool described?:Tool details:VA results reportedSpecify other:116

Health Technology Assessment 2003; Vol. 7: No. 27Endnote No:Study synthesisIf Yes, how?If quantitativeWas VA considered in study synthesis?Weighted by sample size?Quality threshold definedDefine qual threshold:Extract results?Describe VA method:Summarise resultsResultsPrimary outcomeWhat was the overall conclusion of the review?1:2:3:4:5:6:Describe quantitative results: Statistical measure: Specify otherComparisonPoint estimate95% CI Lower Upper P value00 0 00 0 0 00 0 0 00 0 0 00 0 0 0117© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Appendix 20 0 00Additional outcomes reportedComments118

Health Technology Assessment 2003; Vol. 7: No. 27Author:Journal:Did the review use validity assessment?Did the review consider CMA?Accession No:Endnote No:Year:Were results clearly reported by study design?Was the VA/CMA considered in the study synthesis?Type of review:Aim of intervention:Intervention type:Disease category:Literature searchDatabase1:Database2:Database3:Database4:Database5:Database6:Inclusion criteriaInclusion criteria1:Inclusion criteria2:Languages included:Type/number of studiesOther search1:Other search2:Other search3:Other search4:Other search5:Other search6:No of studies considered:No of studies excluded:Total no of studies:No of RCTs:No of non-RCTs:Total no of patients:No patients:No patients:List type of non-RCTs:No of non-comp:No patients:List type of non-comp:Quality assessmentWas VA carried out?Was independent VA carried out?Same elig criteria?Similar study dates?Was same tool used for all designs?Type of tool:Tool methodSpecify tool:Study designs:Tool described?:Tool details:VA results reportedSpecify other:119© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Appendix 2Endnote NoWas CMA considered at study level?If yes, how?CMA quant statsMethod describedNot reportedVariables describedImpact on reviewWas CMA considered at review level?If yes, what method was used?Which variables were examined?Impact on reviewResultsPrimary outcomeWhat was the overall conclusion of the review?1:2:3:4:5:6:Describe quantitive results: Statistical measure Specify otherComparison Point estimate 95% CI Lower Upper P valueCommentsAdditional outcomes reported?120

Health Technology Assessment 2003; Vol. 7: No. 27Appendix 3Details of identified quality assessment tools121© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

122AuthorOrigin aModified toolTool purpose bStudy designscovered cType of tool d(weighting)Items justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumberAppendix 3Anders, 1996 81 n 7 Coh s e n y 6 na na na na na n nAntczak, 1986 76 m Chalmers, 1981 51 7 RCTs s u n n 30 na na na na na n y 204Audet, 1993 176 m Poynard, 1988 403 7 RCTs, NRS s e n n 17 na 0.70 K (p < 0.01) na na na n y 205Avis, 1994 206 m Fowkes, 1991 107 3 Any c n n 24 na na na na na n nBailar, 1984 207 n 3 Weak/no internal controls c n n 5 na na na na na n nBaker, 1994 82 n 7 Coh c n n NR na na na na na n nBass, 1993 88 n 7 RCTs, NRS; multiple time s u n n 16 na na na na na n nseries and descriptivestudiesBaumann, 1997 208 n 7 RCTs, CTs c n n 20 na na na na na n nBerard, 1997 209 m Chalmers, 1981 51 7 RCTs, prospective CTs s n n NR na na na na na n nBeyer, 1998 210 n 7 RCTs, NRS c n n 4 na na na na na n nBland, 1985 211 m Gifford, 1969 265 2 CCTs c n n 18 na Perfect/good na na na n nexcept item 70.38 K , item 10.07 K , item 6–0.19 KBoers, 1991 212 m Sackett, 1985 349 7 RCTs, Coh c n n 21 na na na na na n nBorghouts, 1998 213 n 7 RCTs, Coh; longitudinal s e n n 13 na na na na na n nstudies; FU; prospectivestudies; retrospectivestudies; CCSBours, 1998 73 m van der Windt, 7 RCTs, CCTs s u n y 18 na 84% a na na na n n1995 71Bracken, 1989 104 n 6 Obs c n n 36 na na na na na n nBrown, 1991 214 m Haynes, 1976 273 7 RCTs, NRS; Pre/Post s u n n 6 na 89% a na na na n n 215continued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.AuthorOrigin aModified toolBrown, 1995 216 n 7 RCTs, NRS c n n 5 na 89% a na na na n nBrown, 1996 217 m Brown, 1991 214 7 RCTs, NRS; Pre/Post s n n 6 na 94.5% a na na na n nCallahan, 1991 218 n 7 RCTs, NRS c n n 9 na na na na na n nCameron, 2000 83 n 7 RCTs, NRS c n n 36 na na na na na n nCameron, 2000 83 n 7 RCTs, NRS; Coh; CCS s u n n 9 na na na na na n nCampos-Outcalt, 1995 177 n 7 RCTs, Quasi-Exp; Coh; s e n n 7 na na na na na n nCS; CCS; correlationalstudiesCanadian Task Force, L 221–2251997, 219 1994 220Cao, 1997 226 m Chalmers, 1981 51 7 RCTs, NRS s n n NR na na na na na n nCarlson, 1996 227 n 7 RCTs, NRS c n n 7 na na na na na n nCarter, 1994 178 m Chalmers, 1981 51 7 RCTs, NRS s u n n 12 na na na na na n nCASP, 1999 64 n 3 Coh c n n 10 na na na na na n nCEBM, 2002 228 L Can Task Force, 41979 219Tool purpose bStudy designscovered cChalmers, 1981 51 n 3 RCT s u n n 36 na na na na na n y 229Cheatham, 1995 230 m Chalmers, 1981 51 7 RCTs, NRS s u n n NR na na na na na n nCho & Bero, 1994 84 m Spitzer, 1990 105 3 Exp, Obs s e n n 31 FC, 0.64 W ; na na na y nCr 0.89 r (CI: 0.73 to0.96)Type of tool d(weighting)Ciliska, 1996 231 n 7 RCTs, Coh, CCS c n n 6 na 0.80 K na na na n n 232Cochrane EPOC n 7 RCTs, CCTs c n n 7 na na na na na n y EPOCGroup, 2002 233reviewsItems justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumbercontinuedHealth Technology Assessment 2003; Vol. 7: No. 27123

124AuthorOrigin aModified toolTool purpose bStudy designscovered cType of tool d(weighting)Items justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumberAppendix 3Cochrane Handbook, X Not a QA tool 234, 2352002 118Cochrane m Chalmers, 1981 51 7 RCTs, CCTs s e n n 12 na na na na na n y CochraneMusculoskeletal InjuriesreviewsGroup, 2002 179Cohen, 1994 236 n 7 RCTs, NRS s e n n 5 na 0.707 K , p = 0.0005 na na na n nColditz, 1994 237 n 7 RCTs, CCS s n y 6 na na na na na n nColeridge Smith, 1999 97 n 7 CCTs, Coh, CCS, CS c n y 49 na C/M: 0.57 K na na na n n(CI: 0.49 to 0.65)C/C: 0.34 K(CI: 0.19 to 0.49)M/M: 0.94 K(CI: 0.71 to 1.0)Cook, 1979 50 X Not a QA tool 189Cook, 1992 238 L 4 y 239–241Cowley, 1995 109 n 7 RCTs, NRS; Obs studies c n n 12– na na na na na n nwithout controls 17Cox, 1989 242 n 7 RCTs, NRS; Obs s u n n 14 na 0.98 K na na na n nCuddy, 1983 180 n 3 Exp c n n 17 na na na na na n nCuijpers, 1998 243 n 7 RCTs, Pre/Post c n n NS na na na na na n nDawson-Saunders, n 3 CCTs, Coh, CCS c n y 49 na na na na na n n1990 98de Craen, 1996 244 m ter Riet, 1990 68 7 RCTs, NRS s u n n 13 na na na na na n nde Kruif, 1996 245 n 7 RCTs, NRS; Coh c n n 8 na na na na na n nde Oliveira, 1995 96 m Chalmers, 1981 51 7 Not clear, CCTs s u n n 30 na Unfamiliar 0.75 r , na na na n y 246familiar 0.87 rcontinued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.AuthorOrigin aModified toolTool purpose bStudy designscovered cde Vet, 1997 72 m ter Riet, 1990 68 7 RCTs, NRS s e n n 15 na 0.77 r na na na n n(0.64–0.89) 67DerSimonian, 1982 247 n 6 CCTs c n n 11 na 9 items > 80% a na na na n nDetsky, 1992 92 m Chalmers, 1981 51 7 RCTs c n n 14 na 0.53 r248 na na na n y 248–249Devine, 1992 250 n 7 Not clear c n n 3? na 89% a , 0.79 K na na na n nDevine, 1983 251 n 7 RCTs, NRS c n n 6 na 92% a na na na n nDevine, 1995 252 n 7 RCTs, Quasi-Exp; Pre/Post c n y NR na 87–100% a na na na n nDevine, 1995 253 n 7 RCTs, NRS c n n NR na na na na na n nDowns, 1998 85 n 3 RCTs, NRS s u n n 27 FC, 0.75 r na 0.88 r na y y 112, 130Cr@2 wks0.89 KrDuffy, 1985 89 n 3 Any s e n n 51 na 0.89 254 0.94 na 0.91 CA n y 254at2 monthsDuRant, 1994 99 n 3 Exp, Quasi-Exp, Survey, CS c n y 103 na na na na na n nDurlak, 1997 255 n 7 RCTs, CTs c n n 6 na 0.85 a (range na na na n n0.75–0.99)Elvik, 1995 256 n 7 NRS c n n NR na na na na na n nEsdaile, 1986 257 n 3 Coh, CCS c n n 6 na na na na na n nFarquhar, 1959 258 n 3 Exp, c n n 18 na na na na na n nFowkes, 1991 107 n 3 CCTs, Coh, CCS, CS c n n 23 na na na na na n y 97Fox, 1958 259 n 3/1 NS c n n 7 na na na na na n nFriedenreich, 1993 100 n 7 Coh, CCS c n y 29 na na na na na n nType of tool d(weighting)Items justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumbercontinuedHealth Technology Assessment 2003; Vol. 7: No. 27125

126AuthorOrigin aModified toolTool purpose bStudy designscovered cType of tool d(weighting)Items justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumberAppendix 3Ganong, 1987 123 X Not a QA tool y 260Gansevoort, 1995 261 n 7 RCTs, NRS c n n 3 na na na na na n nGarber, 1996 262 n 7 Any, CS; Coh; CCS; CCTs s e n n 6 na na na na na n nGardner, 1986 181 n 2 RCTs, NRS c n n 26 na na na na na n nGarg, 1998 263 n 7 RCTs, Coh; CCS c n n 5 na na na na na n nGartland, 1988 124 X Not a QA tool y 264Gifford, 1969 265 n 2 RCTs, NRS c n n 8 na na na na na n nGlantz, 1997 182 m Chalmers, 1981 51 7 RCTs, NRS s u n n 20 na na na na na n nGood, 1996 266 n 7 RCTs, NRS c n n 7 na na na na na n nGordis, 1990 101 n 3 Trials, Coh, CCS, c n y 17 na na na na na n y 267Time series, CSGray, 1982 268 n 2 RCTs, NRS s e n n 8 na na na na na n nGreenhalgh, 1997 183 n 3 RCTs, NRS, Coh, CCS c n n 11 na na na na na n nGurman, 1978 184 n 2 RCTs, NRS s u n n 14 na na na na na n nGuyatt, 1993 79 n 3 RCTs, c n n 12 na 0.79 r 185 na na na n y 185Gyorkos, 1994 127 n 7 CCTs, Coh, CCS, CS, c n y 40 na na na na na n y 269–271Community trials, Obscommunity studies,descriptive studiesHadorn, 1996 102 m US Task Force, 4 RCTs, Coh, CCS, CS and c n y 41 na na na na na n n1989 378 registries, case reports andexpert opinionHancock, 1990–95 272 n 7 RCTs, studies with a c n n 10 na na na na na n ncontrol and treatment groupHayes, 1992 204 X Not a QA tool ncontinued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.AuthorOrigin aModified toolTool purpose bStudy designscovered cHaynes, 1976 273 n 2 RCTs, Quasi-Exp, s u n n 6 na nr na na na n n 274, 275Analytic, DescriptiveHeacock, 1997 276 n 1/3 CCTs s u n n 14 na Range 0.36, 1.00 K ; na na na n nexcellent for 11questions (75–100),good for 2 items(53–64%) and poorfor 1 item (36%)Hedges, 1986 277 X Not a QA tool nHelfenstein, 1994 278 n 7 RCTs, NRS c n n 11 na na na na na n nHeneghan, 1996 279 m Chalmers, 1981 51 7 RCTs, NRS s e n n 15 na 91% a ; 0.83 K na na na n nHill, 1967 280 n 2 RCTs, NRS; Coh c n n 10 na na na na na n nHines, 1969 281 n 3 Clinical investigations c n n 10 na na na na na n nHoag, 1997 282 n 7 RCTs, NRS c n n 29 na 0.88 K n nHoffman, 1993 283 n 7 RCTs, NRS; Coh c n n 9 na na na na na n nHowell, 1997 284 n 7 Not stated c n y 5 na na na na na n nJabbour, 1996 285 n See Klassen, 1995 299 7 RCTs, NRS; Coh s e n n 5 na 87% a , 0.81 K , na na na n nSE=0.10Jadad, 1996 46 n 3/7 RCTs s e n n 3 FC, 0.66 r (CI: 0.53 to na na na n y 286–288Co 0.79); 86% a(0.71 K ) 286Johnston, 1994 289 n 7 RCTs, NRS s e n n 5 na na na na na n n 290Kasiske, 1995 291 n 7 RCTs, CCTs s u n n NR na na na na na n nKasiske, 1998 292 n 7 RCTs, prospective control c n n 5 na na na na na n ngroup study; B/AKasiske, 1993 293 n 7 RCTs, NRS c n n 9 na na na na na n nType of tool d(weighting)Items justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumbercontinuedHealth Technology Assessment 2003; Vol. 7: No. 27127

128AuthorOrigin aModified toolTool purpose bStudy designscovered cType of tool d(weighting)Items justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumberAppendix 3Kay, 1996 186 n 7 RCTs, NRS s e n n 20 na na na na na n nKing, 1984 294 n 3 Drug trials c n n 12 na na na na na n nKing, 1990 295 m King, 1984 294 3 Drug trials c n n 13 na na na na na n nKing, 1990 296 m King, 1984 294 3 Drug trials c n n 17 na na na na na n nKingery, 1997 297 m Koes, 1991 70 7 Not clear, CCTs s u n n 13 na na na na na n nKinney, 1996 298 m Smith and 7 Exp, Quasi-exp s e n n 21 na 0.79–0.94 W na na na n nStullenbarger,1991 360Klassen, 1995 299 m ter Riet, 1990 68 7 RCTs, NRS s u n n 45 na 74% a n nKleijnen, 1994 300 m Kleijnen, 1991 69 7 RCTs, NRS s e n n 10 na na na na na n nKleijnen, 1995 301 m Kleijnen, 1991? 69 7 RCTs, NRS s e n n 10 na na na na na n nKleijnen, 1991 69 m ter Riet, 1990? 68 7 RCTs, NRS s u n n 7 na na na na na n n 302Koes, 1991 70 m ter Riet, 1990 68 7 RCTs, s u n n 17 na 80% a na na na n y 303, 304Kreulen, 1998 187 m Antzcak, 1986 76 7 Not clear, prospective s u n n 17 na 0–1 K ; 0.95 r , na na na n nstudy; cross-CS; p < 0.001retrospective studyKristensen, 1989 305 n 7 NRS c n n 5 na na na na na n y 306Krogh, 1985 307 n 3 Any c n n 11 na na na na na n nKulik, 1990 308 n 7 RCTs, NRS c n y 15 na na na na na n nKwakkel, 1997 188 n 7 RCTs, CCTs s e n n 16 na 0.86 K na na na n nL’Abbe, 1987 120 X No QA tool (Allegedly based on y 309, 402reportedChalmers)Labrecque, 1989 310 n 3 Intervention studies c n n 6 na na na na na n nLeboeuf, 1990 311 n 6/2 RCTs, NRS; Coh; surveys c n y 10 na na na na na n ncontinued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.AuthorOrigin aModified toolTool purpose bStudy designscovered cType of tool d(weighting)Lee, 1997 189 m Cook and Campbell, 7 RCTs, between-subjects s e n n 7 na na na na na n n1979 50 design; within-subjectsdesignLee, 1994 312 n 7 RCTs, NRS s e n n 29 na 94% a na na na n nLevine, 1980 190 n 3 CCTs s e n n 30 na na na na na n y 313Linde, 1999 95 n 2/7 RCTs, NRS s e n n 7 Cr na na na na n y 314Lionel, 1970 315 n 3 Prospective therapeutic c n n 42 na na na na na n ntrialsLittenberg, 1998 316 n 7 RCTs, comparative studies; s e n n 5 na na na na na n nCSLongnecker, 1988 317 n 7 CCS, CCS s u n y 15 na na na na na n nLuborsky, 1975 318 m Fiske, 1970 404 2 RCTs, NRS s e n n 12 na 0.84 r na na na n nLyons, 1991 319 m Smith, 1980 364 7 RCTs, NRS c n n 28 na na na na na n nMacLehose, 2000 26 m Downs and Black, 7 RCTs, NRS; Coh; CCS s n n NR FC, > 60% a na na na n n1998 85 CoMacMillan, 1994 191 m Chalmers, 1981 51 7 RCTs, NRS s u n n 9 na >90% a na na na n nMahon, 1964 320 n 3 Drug trials c n n 4 na na na na na n nMargetts, 1995 321 n 3 Coh, CCS s u n y 31 na CCS 0.71 r ; Similar na na n nCoh 0.92 rMargolis, 1995 322 n 7 RCTs, NRS c n n 6? na na na na na n nNRMarrs, 1995 323 n 7 RCTs, NRS c n n 22 na Categorical na na na n nvariables: 0.891 K(0.625–1);continuous variables:0.933 K (0.693–1)Massy, 1995 324 n 7 RCTs, CTs; uncontrolled s u n y 14 na na na na na n nstudiesItems justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumbercontinuedHealth Technology Assessment 2003; Vol. 7: No. 27129

130AuthorOrigin aModified toolTool purpose bStudy designscovered cType of tool d(weighting)Items justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumberAppendix 3Maziak, 1998 192 n 7 RCTs, NRS s e n n 10 na 0.87 K na na na n nMcAweeney, 1997 325 n 7 RCTs, Quasi-Exp; s e n n 36 na na na na na n ncorrelational studiesMcCusker, 1998 328 m Chalmers, 1981 51 7 Comparative trials, s e n n 6 na 0.84 r (CI: 0.55 na na na n nlongitudinal comparison to 0.95)of at least two groupsMcMaster, 1981 80 n 3 RCTs c n n 6 na na na na na n y 326, 327Meijman, 1995 193 m Fowkes, 1991 107 2 Any s e n n 11 na Mediocre/ Mediocre/ na na n ngood good at15 monthsMeinert, 1986 329 n 2 CCTs c n n 25 na na na na na n nMelchart, 1994 93 n 7 RCTs, NRS s e n n 15 Cr 82% a na na na n nMiller, 1995 87 n 7 RCTs, Matched groups s u n y 12 na 80.7% a na na na n yMoncrieff, 1998 195 n 3 CCTs s e n n 30 na 0.924 r (CI: 0.833 na na 0.873 CA n nto 0.966)Morin, 1994 330 n 7 RCTs, NRS s e n n 6 na na na na na n nMorley, 1996 196 m Chalmers, 1981 51 2/6 RCTs, NRS s u n n 30 na Dichotomous na na na n yvariables 80% a(range 62–98%);continuousvariables:0.91 r (range0.76–0.97)Morris, 1988 331 m Bergin and 2 RCTs, NRS; Coh; CS s e n n 23 na 0.619 K (range na na Design factors, na nLambert,1978 405 0.308–0.824; SD 0.52 a ;0.136) measurementintegrity, 0.64;treatmentvalidity, 0.25continued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.AuthorOrigin aModified toolTool purpose bStudy designscovered cType of tool d(weighting)Mullen, 1992 274 m Haynes, 1976 273 7 RCTs, NRS c n n NR na 90.6% a na na na n nMulrow, 1986 197 n 7 RCTs, NRS; Coh c n n 13 na 94% a na na na n nMurtagh, 1995 332 n 7 Any c n y 17 na 95% a na na na n nNaldi, 1990 333 m Chalmers, 1981 51 2 RCTs, NRS; Coh c n n 8 na na na na na n nNaylor, 1987 334 m Chalmers, 1981 51 7 RCTs, NRS s n n NR na na na na na n nNewcastle-Ottawa 66 n 7 Coh, CCS s u n y 8 na na na na na n nNicolucci, 1989 77 m Chalmers, 1981 51 7 RCTs s u n n 16 na 88% a na na na n y 335Nielsen, 1985 336 n 3 Any c n n 24 na na na na na n nNyberg, 1974 337 n Jonsson, 1969 406 1/3 CCTs s e n n 11 na na na na na n nOakley, 1994 128 n 7 RCTs, CTs c n n 8 na na na na na n y 338–340Ogilvie-Harris, 1995 341 m Weiler, 1992 388 7 RCTs, CCTs s e n n 10 na na na na na n nOsberg, 1997 342 n 2 Outcome research c n n 8 FC na na na na n nOxman, 1994 343 n 7 RCTs, NRS c n n 6 na na na na na n nPowe, 1994 344 n 7 NRS, s e n y 11 na 93.5 K na na na n nPuig Barbera, 1995 345 n 7 RCTs, Coh c n n 8 na na na na na n nReisch, 1989 111 n 1/3 Therapeutic studies s e n n 57 na 0.99 P for objective na na na n y Cochranescore; 0.71 P forIBDsubjective score 114GroupRey, 1997 346 n 7 RCTs, Coh or case reports s n n 5 na na na na na n nRied, 1994 347 m Chalmers, 1981 51 7 RCTs, CCTs s e n n 9 na na na na na n nRowe, 1997 348 n 7 RCTs, NRS; uncontrolled s n n 12 na na na na na n nItems justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumbercontinuedHealth Technology Assessment 2003; Vol. 7: No. 27131

132AuthorOrigin aModified toolTool purpose bStudy designscovered cType of tool d(weighting)Items justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumberAppendix 3Sackett, 1989 126 , 1985 349 L 4 y 350, 351Salisbury, 1997 198 n 7 RCTs, descriptive studies; c n n 45 na na na na na n nmethodological studiesSchechter, 1991 199 n 3 CCT c n n 30 na na na na na n nSchmid, 1991 121 X Not a QA tool y 352Sellers, 1997 353 n 7 RCTs, NRS c n n 13 na na na na na n nSelley, 1997 354 n 7 NRS, Coh s n n 9 na na na na na n nShaver, 1989 355 n 7 RCTs, Quasi-Exp, Pre/Post c n n 162 na 94% a na na na n nShay, 1972 90 n 3 Any c n n 25 Fa high na na na n nSheldon, 1993 200 n 3 RCTs, Quasi-Exp, Coh, c n n 36 na na na na na n nCCS, CSSimons, 1996 356 n 7 RCTs, CCTs s e n n 9 na 85% a na na na n nSlavin, 1986 357 X Not a QA tool y 358Smeenk, 1998 359 n 7 RCTs, CCTs s u n n 34 na 90.8% a na na na n nSmith, 1991 360 n 7 Exp, Non-Exp s e n n 22 na na na na na n ySmith, 1977 361 X Not a QA tool y 362Smith, 1988 125 X Not a QA tool y 363Smith, 1980 364 n 7 RCTs, CCTs c n n 21 na 92% a na na na n nNRSolomon, 1993 365 m Evans and Pollock, 2 RCTs, NRS; case studies s n n 10 na 0.69 K 0.67 K na na n n1985 407Solomon, 1997 366 n 7 RCTs, Coh; survey c n y 11 na na na na na n ncontinued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.AuthorOrigin aModified toolTool purpose bStudy designscovered cType of tool d(weighting)Spitzer, 1990 105 n 7 RCTs, Coh; CCS; CS; c n n 20 na na na na na n y 97uncontrolled series;descriptive studiesStevens, 1994 367 L US Task Force, 4 y 3681989 378Stock, 1991 369 n 7 NRS, CCS, cross-sectional c n n 7 na na na na na n nstudies; longitudinal CohSuydam, 1968 370 n 3 Exp s e n n 9 na Coefficients 0.91, Coefficients na na n y 3710.94; 0.78 r : 371 0.57, 0.77Talley, 1993 106 n 7 RCTs, CTs c n n 29 na na na na na n nTangkanakul, 1997 372 n Could be from 7 RCTs, NRS c n y 10 na na na na na n nCochrane?ter Riet, 1990 68 n 7 RCTs, NRS s u n n 18 na na na na na n yTheis, 1997 373 n 7 RCTs, comparison studies c n n 5 na na na na na n nThomas, 1991 374 X Not a QA tool y 325Thomas, 1995 375 n 7 RCTs, NRS s u n n 10 na na na na na n nThomas 65 n 7 CCTs, Coh, CCS, B/A, CS c n n 21 na na na na na n nTuckman, 1990 376 n 6? Quantitative research s n n 30 na na na na na n nUS Task Force, 1996, 377 L 379–1989 378 383van Balkom, 1997 384 m Chalmers, 1981 51 7 RCTs, NRS s e n n 14? na na na na na n nNRvan Dalen, 1958 385 n 1/6 Exp c n y 113 na na na na na n nvan der Windt, 1995 71 m Koes, 1991 70 7 RCTs, s u n n 17 na 90% a na na na n y 73Items justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumbercontinuedHealth Technology Assessment 2003; Vol. 7: No. 27133

134AuthorOrigin aModified toolTool purpose bStudy designscovered cType of tool d(weighting)Items justified?Items by design?No. of itemsTool validity eInter-raterreliability fIntra-raterreliability fTest-retest fInternalconsistency fTime to completeUsed in otherreviews?Review referencenumberAppendix 3Vickers, 1995 110 n 3 CCTs c n n 21 na na na na na n nVickers, 1994 386 n 7 RCTs, Coh c n n NR na na na na na n nVickers, 1996 201 n See also Vickers, 7 RCTs, NRS c n n 12 na “good” na na na n n1994 386Ward, 1979 387 n 3 Any c n n 53 na na na na na n nWeiler, 1992 388 n 7 RCTs, CTs c n n 14 na na na na na n yWeintraub, 1982 108 m Lionel, 1970 315 1/2/3 CCTs c n n 47 na na na na na n nWells-Parker, 1995 129 n 7 RCTs, NRS s e n n ? FC grouping strategies na na na n ndomain: 0.95 rWilson, 1992 103 n 7 RCTs, Coh, CCS c n y 10 na na na na na n nWingood, 1996 202 n 7 RCTs, NRS c n n 16 na na na na na n nWolf, 1993 389 n 5 Any s e n n 31 na na na na na n nWoolf, 1990 390 L Can Task Force, 4 y 3911979 219Wright, 1995 203 n 7 RCTs, NRS c n n 13? na na na na na n nZaza, 2000 86 n 7 RCTs, NRS c n n 22 na na na na na n nZola, 1989 392 m Chalmers, 1981 51 2 RCTs, NRS; Coh c n n 11 na na na na na n na Origin: n, new; m, modified; X, not a quality assessment tool; L, levels of evidence classification system.b Tool purpose: 1, planning a study; 2, assessing a statistical/methodological analysis; 3, evaluating a study when considering the practical application of the intervention; 4, evaluatingor grading study recommendations when producing guidelines; 5, peer reviewing; 6, reporting a study; 7, assessing studies included in a systematic review (see Box 3).c Study designs covered: RCT, randomised controlled trial; CCT, controlled clinical trial; Coh, cohort; CCS, case–control; Exp, experimental; B/A, before and after; Pre/Post,pre/post; Obs, observational; CS, case series (or cross-sectional); FU, follow-up studies; NRS, non-randomised studies.d Type of tool: c, checklist; s, scale; u, unequal weighting; e, equal weighting.e Tool validity assessments: FC, face and content; Cr, criterion; Co, content; Fa, factorial.f Tool reliability assessments: IRR, inter-rater reliability; IaRR, intra-rater reliability; IC, internal consistency; T-R, test-retest; K, kappa; W, Kendall’s W; a , % agreement; r , intra-classcorrelation; KR, Kuder–Richardson formula 20; CA, Cronbach’s alpha; P, Pearson correlation; α , coefficient alpha; C, clinician; M, methodologist; SE, standard error. NB: low internalconsistency suggests that the criteria are relatively easy to satisfy.na, not assessed; NR, not reported.

Health Technology Assessment 2003; Vol. 7: No. 27Appendix 4Detailed description of ‘unselected’top 14 quality assessment toolsBracken, 1989 104The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredIf historical rather than concurrent controlshave been used, has it been explained whythis does not introduce bias into the study?5.4 Any attempt to balance groups by designIf study groups were ‘matched’, have therationale and detailed criteria for matching,and success (i.e. number of cases not matched)been provided?9.2 Identification of prognostic factorsHas the measurement of importantconfounding or effect-modifying variablesbeen described so the reader can judge howthey have been controlled?9.3 Case-mix adjustmentHas an analysis of potential confounding oreffect modification of the principal relationsbeen presented?This tool provides a list of 36 questions to guidethe reporting of observational studies. The tool issplit into four sections: introduction to the report(4 items); description of materials and methods(17 items); presentation of results (7 items); andstudy conclusions (8 items). The tool took 20–30minutes to complete. It was clearly not designedfor use in a systematic review and to ourknowledge has not been used for that purpose.The majority of the questions could be answeredyes/no, making it possible to compare responsesacross studies. However, the questions are notphrased in such a way that any real judgements ofthe methodological quality of a study could bemade, as the questions listed above demonstrate.Furthermore, the study designs that the tool couldcover were limited to case-control and cohortdesigns.Critical Appraisal SkillsProgramme, 1999 64The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredItem not covered5.4 Any attempt to balance groups by designHave the authors taken account of theconfounding factors in the design and/oranalysis?9.2 Identification of prognostic factorsWhat factors could potentially be related toboth the exposure and the outcome of interest(confounding)?9.3 Case-mix adjustmentSame item as 5.4 aboveThis tool provides a list of 12 questions to aid thecritical appraisal of cohort studies. The tool is splitinto three sections: ‘are the results of the studyvalid?’ (7 items); ‘what are the results?’ (2 items)and ‘will the results help locally?’ (2 items). Thetool took 15–20 minutes to complete. It was notused in our sample of systematic reviews, and in itscurrent format would be difficult to apply in sucha context. The questions prompt thinking aboutquality but were felt to require subjective answersthat are unlikely to be consistent within or betweenreviewers. The questions are followed by ‘hints’ tohelp the reader answer the overall questions, butin some cases it is difficult to work out how theresponses to the hints translate into yes/no/can’ttell. For example, if a reviewer answered ‘yes’ totwo hints and ‘no’ to two hints for a singlequestion, it is not clear what the overall answer tothat question should be.The CASP tool was not judged to be suitable foruse in a systematic review in its current format.The Bracken tool was not judged to be suitable foruse in a systematic review.135© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Appendix 4136DuRant, 1994 99The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredHow were subjects assigned to experimentalgroups?5.4 Any attempt to balance groups by designItem not covered9.2 Identification of prognostic factorsAs for 9.3 below9.3 Case-mix adjustmentWere appropriate variables or factorscontrolled for or blocked during the analysis?Were other potentially confounding variableshandled appropriately?This tool provides a list of 103 questions to aid theevaluation of research articles. The tool coversseveral study designs: experimental or quasiexperimental,survey designs and cross-sectionalstudies, retrospective chart reviews andretrospective studies and case–control studies.Some of the sections are general across all designs:‘introduction’ (8 items); ‘methods and procedures’(3 items); ‘results’ (12 items); and ‘discussion’(9 items). Additional sections relevant to nonrandomisedintervention studies are:‘experimental or quasi-experimental designs’(26 items) and ‘statistical analysis for experimentaldesigns’ (5 items). The tool took 15–25 minutes tocomplete. Again, it was not designed for use in asystematic review and although it covers themajority of issues relating to internal validity, thetool does not force the reader to answer thequestions in a systematic manner. It is also verylong and includes several irrelevant items. Thistool is really a critical appraisal tool designed toprompt thinking regarding quality and does notprovide a means of comparing the quality ofstudies included in a review.The DuRant tool was not judged to be suitable foruse in a systematic review.Fowkes and Fulton, 1991 107The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredItem not covered5.4 Any attempt to balance groups by designControl group acceptable? Matching orrandomisation9.2 Identification of prognostic factorsDistorting influences? Confounding factors9.3 Case-mix adjustmentDistorting influences? Distortion reduced byanalysisThis tool provides a list of six questions as achecklist for appraising a medical article. The toolaims to cover several study designs includingcross-sectional, cohort, controlled trials andcase–control studies. Each of the six questions listsitems to be scored ‘major problem’, minorproblem’ or ‘no problem’. The tool took around10 minutes to complete. This is another exampleof a checklist that was designed to promptthinking about quality but does not permit thesystematic assessment of quality across studies.Limited detail of what the items mean is providedin the actual checklists and the entire paper needsto be read to provide any guidance.The Fowkes tool was not judged to be suitable foruse in a systematic review.Hadorn and colleagues, 1996 102The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredFor cohort studies, the study groups were nottreated concurrently5.4 Any attempt to balance groups by designItem not covered9.2 Identification of prognostic factorsKnown prognostic factors for the outcome ofinterest or possible confounders were notmeasured at baseline9.3 Case-mix adjustmentA significant difference was found in one ormore baseline characteristics that are knownprognostic factors or confounders, but noadjustments were made for this in the analysisThis tool was developed for the rating of evidencefor clinical practice guidelines. It provides a list ofeight quality assessment criteria; each criterion listswhat the authors consider to be major and minorflaws. The criterion for allocation of patients totreatment groups has separate responses for RCTsand cohort or registry studies. The tool took 15–20minutes to complete. Although the tool covered anumber of validity issues, it was found to be fairlydifficult to use in its published format and perhapsconcentrated overly on pharmaceutical interventions.The Hadorn tool was not judged to be suitable foruse in a systematic review.

Health Technology Assessment 2003; Vol. 7: No. 27Spitzer and colleagues, 1990 105The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredSuitable assembly of comparison group5.4 Any attempt to balance groups by designKnown confounders accounted for by designAny methods to attempt comparabilitybetween groups, other than randomisation9.2 Identification of prognostic factorsSample size enables adequate precision insecondary variables reported (confoundingvariables or incidental findings)9.3 Case-mix adjustmentKnown confounders accounted for by analysisThis tool was developed for use in anepidemiological systematic review. It contains 20quality items and takes 10–20 minutes to complete.It was used as a quality assessment tool in one ofour sample of systematic reviews. 97 It scoresreasonably well on the required quality items andcan be applied to different types of study design;however, some of the questions are rather subjectiveand/or ambiguous (for example, ‘Selection biasaccounted for’) and some quality issues appear tobe covered by more than one item. Very littleguidance for completion of the tool is provided.Several response options are provided, but it isdifficult to differentiate these from each other (e.g.options include Uncertain/Incomplete/Substandard,Don’t know/Not reported, N/A and N/C).Overall, the Spitzer tool was not judged to besuitable for use in a systematic review.Vickers, 1995 110The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredItem not covered5.4 Any attempt to balance groups by designAdequacy of baseline matching9.2 Identification of prognostic factorsAll relevant variables included in baselinematching9.3 Case-mix adjustmentStratification/multivariate analysis adequateand appropriate?This checklist is a tool for critical appraisal,containing 21 items, several of which havefurther sub-questions. The tool takes up to20 minutes to complete. It covers a lot ofvalidity issues, but some of the wording israther ambiguous. This tool was designed toaid critical appraisal and prompts thinkingabout quality rather than providing a tool forsystematic quality assessment. It would bedifficult to use within the context of a systematicreview.The Vickers tool was not judged suitable for use ina systematic review.Weintraub, 1982 108The types of questions which led to the selectionof this tool according to our pre-selected corecriteria were:5.3 How allocation occurredConcurrent or historical controls5.4 Any attempt to balance groups by designAssignment of treatment: randomised,matched or stratification or minimisation9.2 Identification of prognostic factorsComparability of treatment groups on specificcriteria9.3 Case-mix adjustmentItem not coveredThis tool was developed to evaluate criticallyreports of clinical drug trials and to aid in thewriting of protocols and papers. It contains 47items split into seven categories: population,treatments, experimental design details,compliance, data collection, control of bias anddata analysis. The checklist takes 10–15 minutes tocomplete and is aimed at studies of any designthat evaluate drug therapy. The checklist is notstructured in such a way that responses to thequestions are systematic and the items do notreally force the reader to make any qualityjudgements. There is a final section where anoverall assessment of study quality should bemade; however, it is not clear how thisassessment should follow from the precedingitems.The Weintraub tool is not a suitable qualityassessment tool for use in systematic reviews in itspresent format.137© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Health Technology Assessment 2003; Vol. 7: No. 27Appendix 5Coverage of core items by selected top toolsCowley, 1995 1095.3 How allocation occurredMethod of assignment of patients to differentprostheses described5.4 Any attempt to balance groups by designPatient groups matched for diagnoses, age,and illness grade or indicators of activity level,sex, and/or weight, or effect of any differencesevaluated in valid statistical analysis9.2 Identification of prognostic factorsItem not covered9.3 Case-mix adjustmentCovered by question under item 5.4 aboveDowns and Black, 1998 855.3 How allocation occurredWere the patients in different interventiongroups, or were the cases and controlsrecruited from the same population?Were study subjects in different interventiongroups, or were the cases and controlsrecruited over the same periodof time?5.4 Any attempt to balance groups by designItem not covered9.2 Identification of prognostic factorsAre the distributions of principal confoundersin each group of subjects to be comparedclearly described?9.3 Case-mix adjustmentWas there adequate adjustment forconfounding in the analyses from which themain findings were drawn?Newcastle–Ottawa tool 665.3 How allocation occurredItem not covered5.4 Any attempt to balance groups by designComparability of cohorts on the basis ofdesign or analysis9.2 Identification of prognostic factorsStudy controls for most important prognosticfactors9.3 Case-mix adjustmentComparability of cohorts on the basis ofdesign or analysisReisch and colleagues, 1989 1115.3 How allocation occurredSeveral items pertaining to allocation wereincluded, such as, ‘historical controls’ or‘convenience (subjects selected foravailability)’ and ‘other non-randomisedmethod’5.4 Any attempt to balance groups by designUse of either prognostic stratification prior tostudy entry or retrospective stratificationduring data analyses9.2 Identification of prognostic factorsItem not covered9.3 Case-mix adjustmentUse of either prognostic stratification prior tostudy entry or retrospective stratificationduring data analysesThomas 655.3 How allocation occurredItem not covered5.4 Any attempt to balance groups by designIndicate the percentage of relevantconfounders that were controlled in thedesign9.2 Identification of prognostic factorsWere there important differences betweengroups prior to the intervention (a list ofimportant confounding factors wasprovided)?9.3 Case-mix adjustmentIndicate the percentage of relevantconfounders that were controlled in theanalysisZaza and colleagues, 2000 865.3 How allocation occurredItem not covered139© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Appendix 55.4 Any attempt to balance groups by designConsidering the study design, wereappropriate methods for controllingconfounding variables and limiting potentialbiases used?9.2 Identification of prognostic factorsDid the authors identify and discuss potentialbiases or unmeasured/contextual confounders,etc.?9.3 Case-mix adjustmentAs 5.4 above140

Health Technology Assessment 2003; Vol. 7: No. 27Appendix 6Details of review methods141© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

142Author Review IV type b Topic Total RCTs NRS Other Type of QA Tool QA results dtype atool cACCP/AACVPR, Nar Rehab Pulmonary disease 76 49 ? 0 LoE US Task Force 377,378 Nar1997 393Anders, 1996 81 MA Vacc Measles 10 0 10? 0 S Author’s own Mean scoreAronson, 1996 249 MA Pharm Depression 8 4 4 0 S Detsky, 1992 92 Itemised; NarAudet, 1993 176 Nar Educ Medical students 10 0 10 0 S Modified Poynard, 1988 403 Itemised; OSBaker, 1994 82 Nar Surgery Coronary disease 38 0 7 31 C Author’s own ItemisedBass, 1993 88 Nar Behav Childhood injury 20 7 10 3 S Author’s own OS; ranked by scoreBauman, 1997 208 Nar Psych Chronic conditions 15 11 4 0 C Author’s own Itemised (most items)Berard, 1997 209 MA Physio Osteoporosis 18 5 13 0 S Modified Chalmers, 1981 51 Mean score and rangeBeyer, 1998 210 MA Vacc Influenza 24 22 2 0 S Author’s own ItemisedBoers, 1991 212 Nar Pharm Rheumatoid arthritis 7 6 1 0 C Modified Sackett, 1985 349 ItemisedBours, 1998 73 Nar Org Aftercare of elderly 17 6 11 0 S Modified van der Windt, Itemised and OS1995 71Brown, 1988 254 MA Educ Diabetes 47 NS NS 0 C Duffy, 1985 89 Mean score; no. with13 threats to validitypresented in Brown,1990 215Brown, 1990 215 MA Educ Diabetes 82 NS NS 0 S Brown, 1991 214 Not reportedBrown, 1995 216 MA Org Primary care 53 14 39 0 C Author’s own Not reportedBrown, 1996 217 MA Mixed Diabetes 89 ? ? 0 S Modified Brown, 1991 214 Mean score and rangeBurckhardt, 1987 362 MA Psych Elderly 41 23 18 0 C Smith and Glass, 1977 361 NarCallahan, 1991 218 MA Pharm Stable chronic obstructive 14 11 4 0 C Author’s own Itemisedpulmonary diseaseCallahan, 1995 264 MA Surg Knee arthroplasty 64 0 64? 0 NR Gartland 1988 124 Not reportedCameron, 2000 83 MA Rehab Fractures in older people 41 14 27 0 S Author’s own OSCamma, 1996 335 MA Pharm Hepatitis C 9 5 4 0 S Nicolucci, 1989 77 Mean scoreCao, 1997 226 MA Pharm Bancroftian filariasis 15 9 6 0 S Modified Chalmers, 1981 51 Not reportedCarlson, 1996 227 MA Occ ther Well-being of older persons 14 6 8 0 C Author’s own NarCarter, 1994 178 Nar Alt med Pre-menstrual syndrome 14 ? ? 0 S Modified Chalmers, 1981 51 OScontinuedAppendix 6

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.Author Review IV type b Topic Total RCTs NRS Other Type of QA Tool QA results dtype atool cCatre, 1997 222 Nar Surg Spinal surgery 15 1 6 8 C Canadian Task Force Categorised by level of1994 220 evidenceChaulk, 1998 240 Nar Other Pulmonary tuberculosis 27 5 3 22 NR Cook, 1992 238 Level of evidenceCheatham, 1995 230 MA Surg Nasogastric decompress 26 15 2 9 S Modified Chalmers, 1981 51 OSCheek, 1998 130 Nar Surg Groin hernia 71 45 26 0 S Downs, 1996 112 ItemisedChoi, 1995 352 MA Pharm Bed nets for malaria 10 7 3 0 S Schmid, 1991 121 Not reportedCiliska, 1995 351 Nar Pharm Obesity in NIDDM e 10 8 2 0 C Sackett and Haynes, Classed strong/1985 349 moderate/weakCiliska, 1996 231 Nar Org Home visiting 77 NS NS 0 C Author’s own ItemisedCohen, 1994 236 Nar Educ Low back pain 13 9 4 0 S Author’s own No. scoring

144Author Review IV type b Topic Total RCTs NRS Other Type of QA Tool QA results dtype atool cDe Oliveira, 1996 246 MA Pharm Schizophrenia 18 ? ? 0 S de Oliveira, 1995 96 NR; all scored above acertain thresholdDetsky, 1987 309 MA Diet Surgical patients 18 11 7 0 S L’Abbe, 1987 120 OSDevine, 1992 250 MA Psych Surgical patients 173 119 54 0 C Author’s own Not reportedDevine, 1983 251 MA Psych Postsurgical patients 49 37 12 0 C Author’s own No. meeting certaincriteriaDevine, 1995 252 MA Psych-ed Hypertension 88 51 37 0 NR Author’s own No. using each methodof assignmentDevine, 1995 253 MA Psych Cancer 98 67 31 0 C Author’s own Not reportedDodhia, 1998 368 Nar Pharm Pertussis transmission 13 2 4 7 LoE Stevens and Raftery, 1994 367 Not reportedDowns, 1996 112 Nar Surg Stress incontinence 76 11 65 0 S Downs, 1998 85 ItemisedDurlak, 1997 255 MA Prim prev Mental health 177 108 69 0 C Author’s own No. meeting eachcriteria reportedEastwood, 1996 394 Nar Org Asthma care 27 9 4 14 C CRD guidelines 119 NarEPI Centre, 1996 338 Nar H promo Sexual health 110 34 NS NS C Oakley, 1994 128 No. each criteriareportedErnst, 1998 287 Nar Alt med Dental pain 16 11 5 0 S Jadad, 1996 46 Itemised; OSErnst, 1998 302 Nar Alt med Muscle soreness 8 3 5 0 S Kleijnen, 1991 69 OSFiscella, 1995 267 Nar Other Pregnancy 50 11 12 C Gordis, 1990 101 NarFlint, 1995 326 Nar Respite Dementia 4 2 2 0 C McMaster, 1981 80 NarcareFloyd, 1997 371 MA Psych Mental health 25 ? ? 0 S Suydam, 1968 370 Not reportedFoxcroft, 1997 340 Nar Psych Alcohol misuse 33 ? ? 0 C Oakley, 1994 128 % meeting eachcriterion reportedGam, 1995 229 MA Physio Musculoskeletal disorders 22 9 13 0 C Chalmers, 1981 51 % meeting eachcriterion reportedGansevoort, 1995 261 MA Pharm Hypertension 41 20 21 0 C Author’s own ItemisedGarg, 1998 263 Nar Pharm Myocarditis 7 3 4 0 C Author’s own ItemisedGlantz, 1997 182 MA Other Obstetrics 5 2 3 0 S Modified Chalmers, 1981 51 Itemised; OSAppendix 6continued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.Author Review IV type b Topic Total RCTs NRS Other Type of QA Tool QA results dtype atool cGood, 1996 266 Nar Psych Postoperative pain 21 11 8 2 C Author’s own NarGriffiths, 1997 225 MA Other Ulcerative colitis 12 0 11 1 LoE Canadian Task Force,1994 220Grimes, 1997 383 Nar Pharm Pregnancy termination 27 7 3 17 LoE US Task Force, 1996 377 Categorised by level ofevidenceGrullon, 1997 382 Nar Org Obstetrics 18 5 10 13 LoE US Task Force, 1996 377 NarGyorkos, 1994 269 MA Vacc Various 54 11 43 0 C Gyorkos, 1994 127 OS; classed strong/moderate/ weakHancock, Nar H promo N/A 13 3 9 1 C Author’s own Itemised1990–1995 272Hayes, 1992 204 Nar Pharm Periodontitis 13 9 4 0 S Antczak, 1986 76 % meeting eachcriterion reportedHearn, 1998 395 Nar Pall care Cancer 18 5 14 0 LoE Cancer Guidance (US Classified by level ofTask Force? 377 )evidenceHelfenstein, 1994 278 MA Dental Caries prevention 8 8 4 0 C Author’s own ItemisedHeneghan, 1996 279 Nar Other Social work 10 5 5 0 S Modified Chalmers, Itemised; OS1981 51Heyland, 1993 350 Nar Diet Critically ill 39 ? ? ? LoE Sackett, 1989 126 Some categorisationby level of evidenceHoag, 1997 282 MA Psych Dysfunctional behaviour 56 NS NS 0 C Author’s own No. meeting eachcriteria reportedHoffman, 1993 283 Nar Surg Herniated lumber discs 81 2 17 62 C Author’s own No. with each criterionreported; categorisedby level of evidenceHowell, 1997 284 MA Behav/ Cholesterol reduction 224 ? ? 0 C Author’s own Nar; % low/medium/Educhigh qualityJabbour, 1996 285 Nar Educ Life support 17 5 8 4 S Author’s own OSJohnston, 1994 289 Nar Org Clinician performance 28 24 4 0 S Author’s own Itemised; OSKasiske, 1995 291 MA Pharm Hypertension 474 NS NS NS Not Author’s own Not reportedreportedKasiske, 1993 293 MA Pharm Renal transplant 21 12 9 0 C Author’s own NarKasiske, 1998 292 MA Diet Renal function 24 13 11 0 C Author’s own ItemisedcontinuedHealth Technology Assessment 2003; Vol. 7: No. 27145

146Author Review IV type b Topic Total RCTs NRS Other Type of QA Tool QA results dtype atool cKay, 1996 186 MA Educ Oral health 37 7 30 0 S Author’s own Not reportedKingery, 1997 297 MA Pharm Pain syndromes 50 NS NS NS S Modified Koes, 1991 70 OSKinney, 1996 298 MA Mixed Cardiac patients 84 54 30 0 S Modified Smith, 1991 360 Mean/modal score andrangeKlassen, 1995 299 Nar Pharm Depression in Parkinson’s 12 9 3 0 S Modified ter Riet, 1990 68 Itemised; OSdiseaseKleijnen, 1995 301 Nar Other Multiple sclerosis 14 11 3 0 S Author’s own Itemised; OSKleijnen, 1994 300 Nar Alt med Cancer 11 6 5 0 S Author’s own Itemised; OSKleijnen, 1991 69Kleijnen, 1991 69 Nar Alt med Any 107 68 39 0 S Author’s own OS; Itemised betterquality onesKrause, 1998 306 Nar Rehab Occupational health 21 1 6 6 S Kristensen, 1989 305 Nar; OSKreulen, 1998 187 MA Dental Veneer restorations 14 14 S Modified Hayes, 1992 204 OS; Mean/modal scoreper itemKrywanio, 1994 363 MA Other Neonates in NICU 39 25 14 0 S Smith, 1988 125 OS; mean score and SDKulik, 1990 308 MA Educ N/A 103* 28 75 0 C Author’s own No. meeting eachcriterion reportedKwakkel, 1997 188 MA Rehab Stroke 9 8 1 0 S Author’s own Itemised; NarLee, 1997 189 MA Other Seasonal affective disorder 40 ? ? 0 S Cook, 1979 50 Not reportedLee, 1994 312 Nar Psych Divorce 15 11 4 0 S Author’s own % meeting eachcriterion reportedLijesen, 1995 303 Nar Pharm Obesity 24 14 10 0 S Koes, 1991 70 Itemised; OSLinde, 1998 314 MA Alt med Various 32 23 9 0 S Linde, 1999 95 Itemised; OSLittenberg, 1998 316 MA Surg Tibial shaft fracture 19 13 S Author’s own Mean/median scoresand rangeLoblaw, 1998 223 Nar Mixed Spinal cord compression 24 ? ? ? LoE Canadian Task Force, Categorised by level of1994 220 evidenceLong, 1995 380 Nar Other Premature infants 31 11 20 0 LoE US Task Force, 1989 378 Categorised by level ofevidenceLucassen, 1998 286 MA Mixed Infantile colic 27 NS NS 0 S Jadad, 1996 46 OSAppendix 6continued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.Author Review IV type b Topic Total RCTs NRS Other Type of QA Tool QA results dtype atool cLyons, 1991 319 MA Behav Psychological conditions 70 NS NS 0 C Modified Smith, 1980 364 % of comparisons ineach categoryMacLehose, 2000 26 MA Screening/ Breast cancer/neural 34 12 11 11 S Modified Downs, 1998 85 Mean score per groupother tube defects of studies according to4 dimensions of biasMacMillan, 1994 396 Nar Psych Child sexual abuse 19 14 5 0 S Modified Chalmers, 1981 51 ItemisedMacMillan, 1994 191 Nar Social Child phys abuse/neglect 11 6 5 0 S Modified Chalmers, 1981 51 ItemisedworkMargolis, 1995 322 Nar Org Discharge of newborns 13 3 10 0 C Author’s own Not reportedMarrs, 1995 323 MA Other Psychological problems 70 59 11 0 C Author’s own % meeting certaincriteria reportedMassy, 1995 324 MA Pharm Renal disease 154 NS NS NS S Author’s own Not reportedMathews, 1996 397 Nar Diet Pregnancy 27 8 19? 0 C Can’t tell NarMayo-Smith, 1997 241 MA Pharm Alcohol withdrawal 65 NS NS 0 C Cook, 1992 238 Not reportedMaziak, 1998 192 Nar Surg Intensive care unit 5 3 2 0 S Author’s own ItemisedpatientsMcAweeney, 1997 325 MA Psych Spinal cord injury 11 0 S Author’s own Mean score per itemMcCusker, 1998 328 MA Mixed Depression in older 37 NS NS 0 S Modified Chalmers, 1981 51 Mean score and SDpeopleMelchart, 1994 93 Nar Alt med Immuno-modulation 26 16 10 0 S Author’s own and Detsky, Itemised; OS1992 92Miller, 1995 87 Nar Mixed Alcoholism 211 159 52 0 S Author’s own ItemisedMorin, 1994 330 MA Behav Insomnia 59 NS NS 0 S Author’s own Not reportedMullen, 1985 275 MA Educ Chronic illness 70 43 27 0 S Haynes, 1976 273 OSMullen, 1992 274 MA Educ Coronary heart disease 38 14 24 0 NR Modified Haynes, 1976 273 No. meeting certaincriteria reportedMulrow, 1986 197 Nar Other Diabetic retinopathy 15 3 1 11 C Author’s own ItemisedMurtagh, 1995 332 MA Psych Insomnia 66 ? ? 0 C Author’s own Not reportedNaylor, 1987 334 MA Pharm Renal failure 5 4 1 0 S Modified Chalmers, 1981 51 OSNHS CRD, 1996 119 Nar Mixed Reduction of injuries 49 22 27 0 C CRD guidelines NarNHS CRD, 1995 391 Nar Public N/A 94 38 56 0 C Woolf, 1990 390 Not reportedhealthcontinuedHealth Technology Assessment 2003; Vol. 7: No. 27147

148Author Review IV type b Topic Total RCTs NRS Other Type of QA Tool QA results dtype atool cNorman, 1998 205 MA Org 10 3 7 0 S Audet, 1993 176 NarOakley, 1995 339 Nar Educ Sexual health 65 20 0 C Oakley, 1994 128 % with certain qualityattributesOakley,1994 128 Nar Educ HIV/AIDS 68 28 40 0 C Author’s own No. meeting certaincriteriaOakley, 1995 234 Nar Behav HIV/AIDS 68 41 27 0 C Cochrane, 1993 118 % meeting eachcriterion reportedOgilvie Harris, Nar Mixed Soft tissue ankle injuries 84 58 14 12 S Modified Weiler, 1992 388 Itemised; OS1995 341Oxman, 1994 343 Nar Org Service delivery 102 NS NS 0 C Author’s own Not reportedOxman, 1994 271 Nar H promo STDs 13 6 7 0 S Gyorkos, 1994 127 OSParker, 1998 398 MA Pharm Ectopic pregnancy 9 0 ? ? NR Modified Chalmers, 1981 51 Not reportedPloeg, 1996 232 Nar Public Adolescent suicide 11 0 11 0 C Ciliska, 1996 231 ItemisedhealthPowe, 1994 399 MA Surg Cataract extraction 90 0 7 83 S Author’s own OSPowe, 1994 344Puig Barbera, MA Vacc Influenza 8 1 7 C Author’s own Itemised1995 345Rey, 1997 346 Nar Psych Psychiatric disorders 60 0 1 59 S Author’s own NarRiben, 1994 270 Nar Educ Public health 13 NS NS 0 S Gyorkos, 1994 127 OSRichard, 1997 224 Nar Org Colorectal cancer 17 4 5 8 LoE Canadian Task Force, Categorised by level1994 220 of evidenceRied, 1994 347 MA Pharm Mountain sickness 20 ? ? 0 S Modified Chalmers, 1981 51 OSRobert, 1997 379 Nar Org Physical therapy 8 1 5 2 LoE US Task Force, 1989 378 Classified by level ofevidenceRowe, 1997 348 MA Surg Idiopathic sclerosis 20 1 0 S Author’s own Not reportedSaint, 1998 288 MA Other Urinary tract infection 8 6 2 0 C Jadad, 1996 46 ItemisedSalisbury, 1997 198 Nar Org N/A 40 7 13 20 C Author’s own NarAppendix 6continued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.Author Review IV type b Topic Total RCTs NRS Other Type of QA Tool QA results dtype atool cSchmidt-Nowara, Nar Other Obstructive sleep apnoea 21 0 21 0 LoE Cook, 1992 238 Categorised by level of1995 239 evidenceSchoemaker, 1997 358 Nar Psych Anorexia nervosa 6 6? 0 C Slavin, 1995 122 ItemisedSellers, 1997 353 MA Educ Public health 7 0 7 0 C Author’s own ItemisedSelley, 1997 354 Nar Surg Prostate cancer unclear 0 C Author’s own Not reportedSimons, 1996 356 MA Surg Inguinal hernia 11 8 3 0 S Author’s own Itemised; OSSmeenk, 1998 359 Nar Pall care Cancer 8 5 3 0 S Can’t tell ItemisedSnowdon, 1997 400 Nar Other Pre-school vision 11 5 6 0 NR Can’t tell Methodologicalshortcomings presentedSolomon, 1997 366 Nar Org Rheumatic/ 16 1 12 3 C Author’s own Itemisedmusculoskeletal conditionsSullivan, 1995 290 Nar Org GP computing 30 19 11 0 S Johnston, 1994 289 Itemised; OSTalley, 1996 401 Nar Psych Irritable bowel syndrome 14 13 1? 0 C Author’s own ItemisedTalley, 1993 106Tangkanakul, 1997 372 MA Pharm Carotid end-arterectomy 20 17 3 0 C Author’s own Narter Riet, 1990 68 Nar Alt med Chronic pain 51 40 11 0 S Author’s own ItemisedTheis, 1997 373 Nar Pharm Neonatal abstinence 14 8 6 0 C Author’s own NarsyndromeThomas, 1995 375 Nar Physio Pulmonary disease/CF 17 2 7 0 S Author’s own OSTowler, 1998 235 MA Other Colorectal cancer 6 4 2 0 C Cochrane Handbook, Itemised1987 118van Balkom, 1997 384 MA Mixed Panic disorder 106 ? ? 0 S Modified Chalmers, 1981 51 Mean scoreVerhagen, 1997 304 Nar Alt med Arthritis 14 7 7 0 S Koes, 1991 70 ItemisedVickers, 1996 201 Nar Alt med Nausea/vomiting 33 25? 8 0 C Author’s own ItemisedVickers, 1994 386 Nar Alt med Irritable bowel syndrome 17 4 5 4 C Author’s own NarVillalobos, 1998 402 MA Pharm Meningitis 12 2 10 0 S L’Abbe, 1987 120 Not reportedWatt, 1998 221 Nar Behav Not clear 11 7 4 0 NR Canadian Task Force, OS; level of evidence1994 220 providedcontinuedHealth Technology Assessment 2003; Vol. 7: No. 27149

150Author Review IV type b Topic Total RCTs NRS Other Type of QA Tool QA results dtype atool cWells-Parker, MA Behav Drink/drive offenders 194 NS NS 0 S Author’s own No. scoring in certain1995 129 rangesWingood, 1996 202 Nar Behav HIV/AIDS 7 5 2 0 C Modified Oakley, 1994 128 Itemised; NarWitlin, 1998 381 Nar Pharm Pre-eclampsia and 18 14 4? 0 LoE US Task Force, 1989 378 OS; level of evidenceEclampsiaWright, 1995 203 Nar Other Obstructive sleep apnoea 23 1 22 0 C Author’s own NarAppendix 6a MA, meta-analysis; Nar, narrative.b Rehab, rehabilitation; Vacc, vaccination; Diet, dietary; Pharm, pharmaceutical; Surg, surgery; Org, organisation; Educ, education; Mixed, several interventions; Psych, Psychiatry;Alt med, alternative medicine; Behav, behavioural therapy; Physio, physiotherapy; Psych-ed, psychoeducational; H promo, health promotion; Occ ther, occupational therapy;other, other types of intervention; Prim prev, primary prevention; Exerc, exercise; Pall care, palliative care.c LoE, Levels of evidence; S, Scale; C, checklist; NR, not reported.d Itemised, itemised per study; Nar, narrative description; OS, overall score.e NIDDM = non-insulin-dependent diabetes mellitus.

Health Technology Assessment 2003; Vol. 7: No. 27Appendix 7Results of identified systematic reviews151© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

152Review Method of incorporating Results of quality investigationquality into synthesisACCP/AACVPR, Qualitative Studies were classified according to level of evidence and recommendations were graded accordingly.1997 393 Methodological issues were discussed narrativelyAnders, 1996 81 Quantitative The t-test of quality score differences was non-significant (p = 0.25). The authors conclude that they cannotSubgroup analysis according to size of effect; reject the null hypothesis; no difference in the quality score between studies existedbetween group differences in mean qualityscores investigatedAronson, 1996 249 Quantitative Pooling all studies indicated a strongly positive effect (RR 2.09, 95% CI: 1.31 to 3.32, p = 0.002). Eliminating oneSensitivity analysis sequentially removing NRS with no control group and then one RCT with a treated control group lowered the relative response togroups of studies with sharedtreatment (RR 1.93, 95% CI: 1.13 to 3.29, p = 0.02). Further elimination of an RCT with possibly contaminatedmethodological deficienciesrandomisation increased the relative response to treatment (RR 2.70, 95% CI: 1.89 to 3.86, p = 0.006), andeliminated the statistical heterogeneity of the results. The three NRS which remained in the analysis were HCTsand therefore likely to have biased resultsAudet, 1993 176 Qualitative Study results were discussed in terms of methodological quality. Only two studies were of sufficiently high qualityto demonstrate a positive effect from teachingBaker, 1994 82 Quantitative Two higher quality studies indicated statistically significant reductions in mortality; of the five others, three hadForest plot used to present study results statistically significant reductions and two were non-significantranked by qualityBass, 1993 88 Qualitative Studies grouped by design, and quality score and overall rank presented: some NRS ranked more highly thanRCTs. Found reasonably high-quality evidence to support the interventionBauman, 1997 208 Qualitative A statement was made regarding the overall quality of included studies. Quality stated to be poor, limiting theinterpretation of resultsBerard, 1997 209 Quantitative The correlation between effect size and quality score (r = –0.07, p = 0.19) or sample size (r = 0.04, p = 0.63)Correlation analysis to investigate ES and were not found to be significantquality scoreBeyer, 1998 210 Quantitative When all studies were pooled, a small, positive, but non-significant effect from influenza vaccine was producedSubgroup analysis according to(RD –0.6, 95% CI: –3 to 1.9). The two poor-quality studies (both of the NRS) showed a much more stronglyquality (high/medium/low)protective effect from the vaccine (RD –9.4, no CIs given), and the medium- and high-quality studies (all RCTs)indicated a much more conservative effect (RD –0.3 and 1.1, respectively, no CIs given)Boers, 1991 212 Qualitative Methodological characteristics were discussed in detail and studies were categorised and discussed according tostrength of evidence. Insufficient evidence was found to make any recommendations regarding drugcombinationsBours, 1998 73 Quantitative Three studies with only positive results had lower quality scores; those with only negative results had similarVisual plot used to map relationship between scores to those with mixed (+ve/–ve) outcomes. Studies with highest scores demonstrated no effect. Five NRSstudy outcome and methodological score scored more highly than any RCTAppendix 7continued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.Review Method of incorporating Results of quality investigationquality into synthesisBrown, 1988 254 Quantitative No significant correlations with unweighted mean effect sizes found. Experimental mortality (attrition) was the38 substantive and methodological only variable significantly correlated with the overall weighted mean ES, r = –0.52, p = 0.002characteristics were investigated in acorrelational analysisBrown, 1990 215 Quantitative Correlation analysis found no relation between attrition rate or rating of research quality and mean ES estimates.Correlational analysis of ES and quality score. Lower-quality studies produced smaller ESs for three of the outcome measures and larger ESs for the other two.Subgroup analyses according to quality For four out of five outcomes the SDs associated with non-equivalent control groups were almost twice as(high/low) and rigour of design for five large as for the other designsoutcome measuresBrown, 1995 216 Not consideredBrown, 1996 217 Quantitative None of the relationships were statistically significant. Strongest relationship was with ES for behavioural strategyCorrelation of quality index and effect sizes (r = –0.46, p = 0.07, n = 16); the higher the quality the lower the ES for mean weight loss. Outcome effectsalso partitioned according to study design: all results were statistically significant for pre-/post-test studies(approx. twofold higher), but for experimental studies, only metabolic control variables were statisticallysignificant. No data presentedBurckhardt, 1987 362 Quantitative High internal validity, high/adequate reliability of the outcome measure, and high reactivity of outcome measuresSubgroup analysis according to high/medium demonstrated larger ES (0.60, 0.59, and 0.80 respectively compared with overall ES of 0.53). No informationvalidity; reliability and reactivity of the was provided regarding the statistical significance of these subgroup analyses. Authors comment that theoutcome measuresignificance of study design characteristics is unclearCallahan, 1995 264 Quantitative No significant correlations were with publication year, mean follow-up, sample size, or % lost to FU.Correlation analysis used to determineif study characteristics were correlatedwith outcomesCallahan, 1991 218 Quantitative Inclusion of lower quality studies marginally increased the overall effect and slightly narrowed the CIs. ResultsOnly studies meeting all quality criteria were not presented for low-quality studies only. Treatment ES was also examined visually in relation to sampleincluded in main analysis; qualifying criteria size for studies meeting all quality criteria and the five studies meeting some criteria. The inclusion of lowerwere sequentially relaxedquality studies did not significantly alter the appearance of the graphCameron, 2000 83 Qualitative Little methodological discussion on included studies; however authors conclude that “the available data havesignificant limitations in quantity and quality and allow only tentative conclusions”, namely that there is limitedevidence for the intervention in terms of hospital admission and residential status following discharge. Mainanalyses focus on comparing RCT and NRS resultsCamma, 1996 335 Qualitative Methodological limitations of studies were discussed. Quality score was not used in the meta-analysis. Authorscomment that quality was reasonably good.Cao, 1997 226 Quantitative Weighting effect size by both quality score and sample size made essentially no difference to the overall results.Quality weightingOnly sample size weighted results presentedcontinuedHealth Technology Assessment 2003; Vol. 7: No. 27153

154Review Method of incorporating Results of quality investigationquality into synthesisCarlson, 1996 227 Quantitative All correlations were non-significant. Less than 1% of the ES variability was associated with the quality of studyCorrelation analysis between ES and six design variable (r 2 = 0.003)potential moderator variables (year ofpublication, study design, residency ofsubjects, mean age, sample size and durationof treatment)Carter, 1994 178 Qualitative Results used to indicate that existing research is methodologically weakCatre, 1997 222 Not consideredChaulk, 1998 240 Not consideredCheatham, 1995 230 Quantitative In most cases, high-quality studies produced higher relative risk estimates, although the magnitude of theSubgroup analysis of high quality studies difference in effect varied. In three cases restriction to high quality studies changed the statistical significance of(15 RCTs and five case–control studies) alone the results; in two cases a non-significant result became statistically significant, and in the other the pooled resultfor 15 outcome variablesbecame non-significantCheek, 1998 130 Qualitative Methodological limitations severely limited the conclusions that could be drawn from the literature, however,some recommendations were madeChoi, 1995 352 Quantitative Studies using a placebo control group generally led to higher pooled effects than those with a no treatmentSubgroup analysis according to use of control group (incidence rate ratios 0.801 vs 0.626, respectively)placebo-control groupWeighting by quality score produced a lower rate ratio compared with the unadjusted results; e.g. for placebo-Presentation of results with and without controlled studies, the rate ratios fell from 0.801 (95% CI: 0.693 to 0.926) to 0.757 (95% CI: 0.611 to 0.937).weighting by quality scoreFor the analysis of permethrin-treated bed nets only (n = 9) the effect of using a placebo control group and ofweighting by quality score were not as strikingCiliska, 1996 231 Qualitative Quality (strong/weak, etc.) was discussed alongside the results of each study. Overall, despite limitations in theresearch, the authors conclude that sufficient high-quality studies are available to demonstrate a positive directionof effectCiliska, 1995 351 Qualitative Weak studies excluded from the review. Some methodological characteristics of included studies were discussed.Authors conclude that the evidence is equivocalCohen, 1994 236 Qualitative Only higher scoring studies were used to ascertain whether findings were related to characteristics of studyparticipants, interventions or settings (n = 6). Little sound evdience was found to support the effect of groupeducation on short-term outcomes (3-month FU), and no evidence of an effect at long-term FU (1 year)Colditz, 1994 237 Quantitative Quality explained 30% of the variation in BCG efficacy in the clinical trials; geographic altitude individuallyRegression analysis to examine quality score explained 41% of the variance. The authors could not determine whether these variables were surrogates forand ESfactors such as quality of FU or presence of non-tuberculous mycobacteria in the population. Quality explained36% of the between-study heterogeneity in the case–control studies, and was the only variable to have anyimpactAppendix 7continued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.Review Method of incorporating Results of quality investigationquality into synthesisCole, 1993 327 Qualitative Quality was not discussed in terms of the results of each study, but the authors do acknowledge that seriouslimitations in trial design and measures were presentCole, 1996 185 Quantitative Quality score did not appear to be associated with the RDAbsolute risk reductions for each individualstudy were plotted against quality score ona forest plotColeridge Smith, Qualitative Results of quality assessment were used to include/exclude studies. Only those with internal validity graded as1999 97 strong or moderate were included in the review of treatments. Recommendations appear to be largely based onthe strength of the evdienceCollingsworth, Qualitative Authors concluded that “despite their methodological shortcomings the conclusions from all 12 studies were1997 260 broadly in agreement and substantiated the findings from the non-empirical work”Cooper, 1994 313 Quantitative Clinical improvement was lower in higher quality studies, but the difference was not significant. In studies ofCompared high and low quality ‘very good’ quality a 75% clinical improvement was observed, compared with 86% for ‘good’ studies and 83%for ‘borderline’ studies (p = 0.17)Cowley, 1995 109 Qualitative Studies classified as A/B/C for each study design but only the results of ‘A’ category studies were discussed. OneRCT, four NRS and 32 uncontrolled studies were given an A rating: key results were discussed. No conclusiveevidence was identifiedCox, 1989 242 Qualitative Studies grouped by quality score (high/medium/low) and results discussed in terms of quality. Three high-qualitystudies were equivocal regarding treatments reviewedCuijpers, 1998 243 Quantitative Small non-significant difference in overall ES was observed between studies with and without a control groupUse of control group investigated in (1.14 and 1.06)subgroup analysisCuratolo, 1998 248 Not consideredde Craen, 1996 244 Quantitative Including the quality score as an additional weighting factor in the analysis did not change the results (data notQuality weightingpresented)de Kruif, 1996 245 Qualitative Results of QA were discussed with reviewers’ conclusions. High-quality studies appeared to show positive effectand low-quality were equivalent. Identified trend of effectiveness of myofeedback not conclusive owing tomethodological limitationsde Oliveira, 1996 246 Quantitative All studies bar one scored >0.6. When the lower-quality study was excluded from the meta-analysis the resultsExcluded lower quality studiesdid not change (no data presented)Detsky, 1987 309 Quantitative Quality scores were negatively correlated with the differences in complication (correlation coefficient –0.56,Correlation analysisp = 0.13) and fatality rates (coefficient –0.51, p = 0.031) between groups, i.e. studies with better-quality scoresshowed a smaller trend in favour of intervention groupcontinuedHealth Technology Assessment 2003; Vol. 7: No. 27155

156Review Method of incorporating Results of quality investigationquality into synthesisDevine, 1992 250 Quantitative No relationship was found between validity and estimates of effect on length of stay, respiratory function,Regression analysis to examine relationship postsurgical pain and psychological distressbetween threats to validity (unpublished,high internal validity, little measurementsubjectivity, placebo control) and estimatesof effectDevine, 1983 251 Quantitative Use of physician blinding to group assignment, higher internal validity and placebo-controlled studiesSubgroup analyses were conducted demonstrated higher ESaccording to use of physician blinding, Random assignment produced lower ES than non-random assignment. The statistical significance of theserandom assignment, high vs low internal differences was not providedvalidity and use of placebo control vs usual The authors conclude that the intervention is effective and does not depend on quality factors inflating the EScare controlDevine, 1995 252 Quantitative ES values for relaxation and blood pressure were negatively correlated with all three measures of qualityMultiple regression analysis between ES andthree measures of quality (random assignment,repeated outcome assessment, post-testmeasurement outside the context ofrelaxation)Devine, 1995 253 Quantitative No significant relationship found for any of the seven dependent variables. No data or results presentedWeighted regression analysis for ES andthreats to validity (unpublished, randomassignment, placebo control)Dodhia, 1998 368 Qualitative Some quality issues discussed narratively alongside individual study results. Overall the level of evidence providedwas judged to be II-2, providing only weak evidence to support the interventionDowns, 1996 112 Qualitative All results were reported individually per study and impact of methodological flaws on results/conclusions ofstudies was discussed. Methodological quality is poor and therefore the value of surgery and the relativeeffectiveness of different procedures is unclearDurlak, 1997 255 Quantitative No significant effects found except for those relating to outcome measures used; normed outcome measures andSubgroup analysis for each quality item, single vs multiple outcome measures lead to lower ESincluding randomisation, placebo control,attrition

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.Review Method of incorporating Results of quality investigationquality into synthesisEPI Centre, 1996 338 Qualitative All studies were discussed in terms of methodology, but only the results of the 21 methodologically sound studieswere discussed in detail. There was disagreement between authors and reviewers regarding effectiveness ofinterventions in 24% of methodologically ‘sound’ studiesErnst, 1998 287 Qualitative Authors focused only on highest quality studies (both scored 3). One found some increase in pain threshold andanother found no effect. Authors’ concluded that there is evidence of benefitErnst, 1998 302 Qualitative Three higher quality studies included (all RCTs); all reported no statistically significant differences; lower qualitytrials indicated some benefit from homeopathyFiscella, 1995 267 Qualitative Evidence was examined in regard to methodological criteria. Authors conclude that current evidence does notsatisfy causal criteria necessary to establish that prenatal birth care definitely improves birth outcomesFlint, 1995 326 Qualitative Some methodological details were provided in the discussion of each individual study. Author concludes thatthere is little evidence for the intervention but that this may be due to methodological limitationsFloyd, 1997 371 Quantitative Not reported in detail; only one variable (use of pretraining) stated to be significantly correlated with effect sizeCorrelation analysis to examine relationshipbetween study and sample characteristics andoutcomeFoxcroft, 1997 340 Qualitative Quality issues discussed along with study results. Most had some methodological shortcomings. No oneintervention could be recommendedGam, 1995 229 Quantitative Impact depended on type of ES used: when d/r was used there was no influence from blinding (p = 0.78), whileSubgroup analysis to examine effect of the d/s showed significant influence (p = 0.009) (d/r and d/s stated to be standardised effect sizes)blindingGansevoort, 1995 261 QuantitativeRegression analysis to examine patient andstudy characteristics and effect on primaryoutcomeGarg, 1998 263 Not consideredStudy design (randomised, double blind or cross-over) made no essential difference to the results. The mostimportant variables were related to the intervention used or clinical characteristics of the populationGlantz, 1997 182 Quantitative Restricting the analysis to only high-quality studies (two RCTs and one NRS) made little difference to theSubgroup analysis of high quality studies only summary effect: the OR increased from 0.61 (95% CI: 0.50 to 0.75) to 0.66 (95% CI: 0.54 to 0.81), indicating alower reduction in the odds of caesarean due to active management of labour. The authors also presentedquality-adjusted results for RCTs only (adjusted by a factor of 2), which further reduced the odds of caesarean:OR 0.70 (95% CI: 0.57 to 0.78). The authors recognise that the larger effect demonstrated when all studieswere combined may be due to the fact all three NRS were HCTs.Good, 1996 266 Qualitative The validity of the results was questioned owing to numerous methodological problems in the studiescontinuedHealth Technology Assessment 2003; Vol. 7: No. 27157

158Review Method of incorporating Results of quality investigationquality into synthesisGriffiths, 1997 225 Qualitative Studies were categorised by level of evidence. The quality of positive European studies is superior to thenegative US studies, so that 13 recommendations could be givenGrimes, 1997 383 Qualitative Studies classified by level of evidence and grade of recommendation. Methodological limitations were notdiscussed in detailGrullon, 1997 382 Qualitative Studies were discussed according to level of evidence provided. Insufficient evidence was found either to supportor to condemn the safety of early postpartum dischargeGyorkos, 1994 269 Quantitative For adult influenza vaccination strong/moderate quality comparisons demonstrated a lower rate difference (RDSubgroup analysis according to internal 17.9%, 95% CI: 16.7 to 19.0%) compared with weak quality comparisons (RD 20.0%, 95% CI: 18.4 tovalidity rating (strong/moderate/weak) 21.6%). For adult pneumococcal immunisation programmes, the average increase in coverage in studies rated‘strong’ (two RCTs and one NRS) was 34.6% (95% CI: –0.6 to 69.8%). Three RCTs and one cohort study ofmoderate internal validity had an average effect of 78% (95% CI: 72.3 to 83.8%). The five ‘weak’ studies(including three RCTs) had a pooled effect of 18.5% (95% CI: 17.7 to 19.3%). The authors made no furthercomment on these results and did not suggest any alternative explanations for these resultsHancock, Qualitative Methodological discussion of included studies – mainly relatively poor. The few methodologically adequate1990–95 272 studies show little evidence of large benefitsHayes, 1992 204 Qualitative Methodological issues were discussed, most studies received low scores. A trend towards effectiveness wasindicated but the authors conclude that this is not clinically significantHearn, 1998 395 Not consideredHelfenstein, 1994 278 Quantitative The only significant explanatory variable was study duration (coefficient of determination r 2 = 0.71, p = 0.002)Correlation analysis between quality itemsand outcome (including study duration,professional cleaning, fluoride rinsing only incontrol group, randomisation, blinded outcomeassessment, double-blinding, attrition)Heneghan, 1996 279 Qualitative Only acceptable studies were included in the review. Methodological assessment used to describe limitations ofthe included studies, rather than to interpret results of the reviewHeyland, 1993 350 Not consideredHoag, 1997 282 Quantitative Experimenter preference for the intervention (identified from introductory section of papers) producedCorrelation analysis used to examine ES and significantly higher ES (0.72 vs 0.30, p = 0.05). Neither the content nor source of outcome measure wereclient, treatment and methodological significantly related to improvement in group treatmentdomainsHoffman, 1993 283 Qualitative Studies examined according to Sackett’s levels of evidence. Both high- and low-quality trials showed positiveeffects from standard discectomy on reoperation rates. Methodological flaws limit the ability to estimate rates ofsuccessful outcomesAppendix 7continued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.Review Method of incorporating Results of quality investigationquality into synthesisHowell, 1997 284 Quantitative No association with outcome was foundRegression analysis to investigate designcharacteristics such as internal validity (high,medium or low), subject location, type ofdiet and the average of multiple observationsJabbour, 1996 285 Qualitative Only studies scoring 3/7 or higher were analysed in detail. Methodological flaws were discussed and in somecases the impact on the interpretation of results was discussed, e.g. quality did not appear to affect the impact onknowledge scoresJohnston, 1994 289 Qualitative Only seven studies scored highly in terms of quality; however, the authors concluded that several sound studiesprovided evidence for at least some of the outcomes evaluatedKasiske, 1995 291 Quantitative A multiple linear regression model was used: few differences were found whether regression was weighted byQuality weightingstudy variance, quality weight or sample size or was unweighted (data not shown). Authors conclude that withrespect to the regression weighting, the results appeared to be robustKasiske, 1998 292 Not consideredKasiske, 1993 293 Quantitative No significant effect was found (results not reported)Subgroup analysis for each of the nine qualityassessment criteria (including randomisation,drop-out rate, selection bias, discontinuationof treatment)Kay, 1996 186 Qualitative Score 12 or more required for inclusion in review, 15 or more for inclusion in meta-analysis. Some discussion ofmethodological problems provided. Narrative review indicated positive effect on knowledge/attitudes. Onlyseven RCTs of high-enough quality to be included in meta-analysisKingery, 1997 297 Not consideredKinney, 1996 298 Quantitative Quality score was associated with ES: higher quality = larger effects. Larger effects also produced by smallerSubgroup analysis according to quality score sample size and convenience sampling. Data not presentedKlassen, 1995 299 Qualitative Studies discussed in terms of methodological flaws; most scored poor to moderate. There is no empiricalevidence available on which to base a treatment plan for depression in patients with Parkinson’s diseaseKleijnen, 1994 300 Qualitative Methodological limitations and implications for study results were discussed. Most trials had shortcomings on fiveor more quality criteria, indicating a serious chance that the results are biased. The results of two ‘highest’ qualitytrials were discussed; however, several shortcomings made it difficult to draw conclusionsKleijnen, 1995 301 Qualitative Results based on highest quality studies; eight higher quality trials indicated that hyperbaric oxygen is notefficacious in the treatment of multiple sclerosiscontinuedHealth Technology Assessment 2003; Vol. 7: No. 27159

160Review Method of incorporating Results of quality investigationquality into synthesisKleijnen, 1991 69 Qualitative Concentrated on results of trials with the best methodological quality. Evidence found to be largely positive,especially in trials of lower qualityKrause, 1998 306 Qualitative Only higher scoring studies were included. A ‘best’ estimate of effectiveness was derived from highest rankingstudies, then a range of probable estimates of effectiveness was derived from the other studies. Authors concludethat there is a suggestion of effectiveness, but methodological limitations make stronger conclusions difficultKreulen, 1998 187 Qualitative Methodological characteristics were discussed narratively; authors conclude that studies were well designed butthat methodological characteristics could be improvedKrywanio, 1994 363 Quantitative:various No significant correlation between ES and quality score regardless of mean birthweight, i.e. the interventionCorrelation analysis of ES and quality score. groups in high-quality studies did not gain more/less than those in lower quality studies.Weighting by sample size, quality score and In one mean birth weight subgroup, only weighting by quality score had an impact on ES [reduced from 1.54mean birth weight in turn(95% CI: 1.22 to 1.86) to 1.37]. Authors comment that this is indicative of poorer quality studies with larger ES.For the other subgroup, only weighting by sample size impacted on ES [reduced from 0.51 (95% CI: 0.39 to0.63) to 0.42], indicating that studies in this group were of higher qualityKulik, 1990 308 Quantitative Random assignment and teacher effects (same or different teachers teaching experimental and control groups)Subgroup analyses according to six quality made little difference to ES or SE. Use of national standardised tests produced lower ESs than locally developedcriteria, further subdivided by teaching tests. Subjectively scored criterion tests produced lower ES in the PSI group compared with objective, machineapproach,i.e. ‘Keller’s Personalised System scoreable exams, but made little difference to the LFM group. Where control groups received the same amountof Instruction’ (largely self-directed learning), of feedback as the intervention groups, the effect size was found to be smaller than where they receivedor Bloom’s ‘Learning for Mastery’standard feedback(teacher-presented and controlled)Kwakkel, 1997 188 Qualitative Methodological limitations of studies were mentioned, and considered as major confounding factors in theconclusionsLee, 1994 312 Qualitative Methodological characteristics discussed separately from results of individual studies; few high-quality studieswere identified; however, some benefits from the interventions were reportedLee, 1997 189 Not consideredLijesen, 1995 303 Qualitative NRS all scored

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.Review Method of incorporating Results of quality investigationquality into synthesisLittenberg, 1998 316 Qualitative Quality and results of each study were discussed individually, starting with highest scoring studies, according touse of control group. Overall quality stated to be poor, but tentative conclusions were drawn from the resultsLoblaw, 1998 223 Qualitative Summary of level of evidence available for each procedure/recommendation. Few high-quality papers were foundLong, 1995 380 Qualitative Studies discussed in terms of level of evidence and grading of recommendations. Authors conclude that a greaterpercentage of studies using indirect positioning as an intervention had stronger evidence to support theinterventions than studies with direct positioning as the intervention. Sackett’s framework should not be the solemethod of treatment evaluation because the infant’s medical status needs to be considered prior toimplementing a treatment strategyLucassen, 1998 286 Quantitative Quality score and effect measure were not correlated (r = –0.02; p = 0.92)Correlation analysis between quality scoreand effectLyons, 1991 319 Quantitative Studies with high internal validity demonstrated larger pooled effect size (ES 1.138, SD 1.119) compared withSubgroup analysis according to quality medium (ES 0.818, SD 0.778) or low (ES 0.811, SD 0.670) validity studies. The low-validity comparison is(high/medium/low) significantly different from the medium and high comparisons (p < 0.05).MacLehose, 2000 26 Quantitative Neither total quality nor any component of the quality score was significantly associated with effect sizeCorrelation analysis to investigate impact ofindividual components of the quality scoreMacMillan, 1994 396 Qualitative Some methodological limitations and their implications on study results were discussed narratively. The impacton the conclusions of the review is not clearMacMillan, 1994 191 Qualitative Methodological limitations were discussed narratively. Study outcomes were also ranked by methodologicalscore. There does not appear to be a great deal of difference in effect according to qualityMargolis, 1995 322 Qualitative Some narrative discussion of quality was provided. Methodological weaknesses make any conclusions regardingeffectiveness impossibleMarrs, 1995 323 Quantitative No-treatment control group led to higher ES than where placebo control used. Researcher contact with subjectsSubgroup analyses according to individual had little impact on ES, and unpublished studies produced slightly higher ES than published, as did randomisedquality criteriacompared with non-randomised studies. A significant correlation was also found between subject retention instudy and ES (r = 0.268)Massy, 1995 324 Quantitative In each case the results of the regression models were not substantively different (data not presented)Regression model weighted by inversevariance and study quality. Results comparedwith weighting by inverse variance alone,sample size and unweighted modelsMathews, 1996 397 Qualitative Studies discussed narratively according to grade (only grades A or B included). Methodological weaknesses makeit difficult to assess the relative importance of each interventioncontinuedHealth Technology Assessment 2003; Vol. 7: No. 27161

162Review Method of incorporating Results of quality investigationquality into synthesisMayo-Smith, 1997 241 Not consideredMaziak, 1998 192 Qualitative Quality issues were discussed alongside the results of each study. A lack of rigorously controlled trials wasidentified and the reviewed evidence provides insufficient information to permit a reasonable conclusion to bemadeMcAweeney, 1997 325 Not consideredMcCusker, 1998 328 Quantitative Results stated to be similar for quality-adjusted and unadjusted effect measures. Only quality-adjusted resultsQuality weightingpresentedMelchart, 1994 93 Qualitative Methodological issues and results discussed. Reviewer’s estimate of strength of evidence provided by each studywas presented. Inconclusive or limited evidence of no efficacy was provided by 21 comparisons and limited orgood evidence of efficacy was provided by 10 comparisonsMiller, 1995 87 Quantitative No significant relation between these codes and methodological quality score was found (r = 0.06). QualityCorrelation analysis of quality score and score was also found to be unrelated to mean problem severity and to estimated cost of delivering the‘outcome logic scores’ (measure of the treatmentstrength of support for treatment efficacy)Morin, 1994 330 Quantitative Design quality was negatively correlated with ES for sleep onset latency (r = –0.24, df = 91, p < 0.03) only,Correlations between outcome and quality indicating that lower quality studies produced larger ESsof study design were calculatedMullen, 1985 275 Quantitative No impact on ES foundRegression analysis to examine effect ofstudy design and type of comparison groupMullen, 1992 274 Qualitative Some narrative discussion of methodological characteristics. Pre-/post-test studies were excluded from the metaanalysisowing to methodological limitations. Further implications of methodological flaws not discussed in any detailMulrow, 1986 197 Qualitative Some quality issues were discussed alongside study results. Authors conclude that methodological limitations donot allow many definitive conclusions but that some new and useful information was obtainedMurtagh, 1995 332 Quantitative Most effect were non-significant – only the source of participants, non-use of drugs and placebo treatmentDesign variables used as inputs into a contributed significantly to the prediction of the dependent variableregression to quantify impact on effect sizes(including source of sample, study design, useof objective validation)Naylor, 1987 334 Quantitative When all studies were combined (3 RCTs and 1 NRS), quality weighting abolished the significant treatment effectSensitivity analysis conducted using weighting for both overall survival from acute episode and overall survival to discharge from hospital. (Inclusion of the NRSby quality score. Weighted and unweighted led to larger treatment effects). Authors conclude that despite the (unweighted) significant findings, there isp-value for the difference between groups insufficient evidence for the interventionwas providedAppendix 7continued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.Review Method of incorporating Results of quality investigationquality into synthesisNHS CRD, 1996 119 Qualitative Only good-quality CTs and case series were presented in the tables. Some methodological comment alsoprovided in the text. It was concluded that the quality of the research needs to be improvedNHS CRD, 1995 391 Not consideredNorman, 1998 205 Qualitative Three studies were rejected owing to low quality; the other seven studies were considered to have relativelyminor methodological problems. The interventions were found to be effective at undergraduate level, but not atresidency levelOakley, 1995 339 Qualitative Only studies judged ‘sound’ were used to evaluate effectiveness. Seven sound studies were reviewed of whichtwo showed short-term effects on reported sexual behaviour. No evidence to indicate that information provisionleads to risk-taking behaviour, but some indication that it may encourage sexual experimentationOakley, 1994 128 Qualitative Characteristics of ‘sound’ studies were discussed in more detail. The effectiveness of interventions was examinedfor both sound and flawed studies. Overall, only a small proportion of interventions were found to have beenevaluated in such a way that conclusions regarding effectiveness could be drawnOakley, 1995 234 Qualitative The conclusions from sound and flawed studies were compared; in general, a larger proportion of sound studiessuggested interventions to be beneficialOgilvie Harris, Qualitative Validity assessment results were presented and discussed but not really used to interpret study results. For1995 341 pharmacological studies around half of the studies were of reasonable quality, providing some evidence that theagents reviewed were better than placebo, but no evidence for one agent over any other. No qualityconsiderations in discussion of other interventions. Studies were examined in regard to the number of statisticallysignificant and non-significant conclusions presented, as there is no standardised method of scoring the severityof ankle injuries or the results of treatmentOxman, 1994 343 Not consideredOxman, 1994 271 Qualitative Quality described separately to results. The numbers of methodologically strong studies which provided data foreach outcome were reported. Overall five studies were found to be methodologically strong. Conclusions arelimited owing to paucity of well-designed studiesParker, 1998 398 Not consideredPloeg, 1996 232 Qualitative Quality features were not discussed in detail; however, authors conclude that the findings of the review shouldbe considered in the light of serious methodological limitations of the studiesPowe, 1994 399 Quantitative No association between visual acuity outcomes and quality score.Quality score and ESFor three complications, higher-quality studies reported higher proportions of complications than studies ofCompared high and low quality lower quality (p = 0.006, 0.02, 0.09)Weighted by quality scoreQuality score was used as a weight in the pooled analysis. Unweighted results were not presentedPuig Barbera, Quantitative Two studies excluded from meta-analysis for not meeting quality criteria; the impact of this on the results of the1995 345 Excluded lower quality studies meta-analysis was not discussedcontinuedHealth Technology Assessment 2003; Vol. 7: No. 27163

164Review Method of incorporating Results of quality investigationquality into synthesisRey, 1997 346 Qualitative No conclusive results available because of poor study quality. Quality improved in the more recent studies.Authors conclude that electroconvulsive therapy in the young is similar in effectiveness and side-effects to its usein adultsRiben, 1994 270 Qualitative Effect of quality on results was discussed. For education: strong/moderate studies indicate no relationshipbetween on-site education and sanitation level. For inspections: strong studies indicate evidence not conclusiveRichard, 1997 224 Qualitative Inconclusive evidence found to support/refute the value of FU surveillance programmes owing to lack of largeand well-designed studiesRied, 1994 347 Quantitative Zero-order correlation (Pearson’s product moment r) was not statistically significant for study ES and qualityCorrelation analysisscore (–0.12), indicating no effect of quality. However, power to detect a statistically significant correlation waslowRobert, 1997 379 Not consideredRowe, 1997 348 Quantitative No significant relationship between quality score and size of treatment effect was found (Pearson’s correlationCorrelation analysis coefficient, r = –0.22, p = 0.33)Saint, 1998 288 Not consideredSalisbury, 1997 198 Qualitative Comment on methodological quality provided in table along with individual study results. Authors conclude thatthere is evidence of benefit from the intervention but that this is based on studies that are methodologicallyweakSchmidt-Nowara,1995 239Not consideredSchoemaker, 1997 358 QualitativeOnly high-quality studies were considered in the review (6/33). Methodological shortcomings and lack of controlfor potential confounders meant that the prognostic value of early intervention could not be estimatedSellers, 1997 353 Quantitative Use of matching led to larger ES for weight and systolic and diastolic blood pressure; use of covariate adjustmentEach coded study characteristic was produced smaller ES for smoking and systolic blood pressure. Longer FU time was associated with larger ES forinvestigated (including sample size, matching, cholesterol, weight and CHD riskresponse rate, covariate adjustment, studysetting)Selley, 1997 354 Not consideredSimons, 1996 356 Quantitative Only high-quality studies included in the meta-analysis (all RCTs ). Summary effect size was larger when all RCTs(including low-quality ones) were includedAppendix 7continued

© Queen’s Printer and Controller of HMSO 2003. All rights reserved.Review Method of incorporating Results of quality investigationquality into synthesisSmeenk, 1998 359 Qualitative Narrative summary of study results and their quality scores was provided. RCTs and NRS appeared todemonstrate similar effects. Effectiveness of the intervention remains unclear.Snowdon, 1997 400 Qualitative Quality issues discusses narratively. Authors conclude there is no evidence of benefit and that most of the studieswere methodologically flawedSolomon, 1997 366 Qualitative Quality issues discussed alongside results of each study. Some indication of differences between care providerswas found, but important methodological limitations remainedSullivan, 1995 290 Not consideredTalley, 1996 401 Qualitative The methodological quality of the studies was discussed in detail; only one study was deemed of acceptablequality; however, its results were not felt to be generalisableTangkanakul, 1997 372 QualitativeMethodological quality discussed separately from study results. Some indication of benefit for the interventionwas found, but too few RCTs and too many problems with NRS to make strong recommendationster Riet, 1990 68 Qualitative Methodological issues were discussed and study results (positive or negative) were examined in relation to thequality of the studies. The authors conclude that no high-quality studies exist and no definitive conclusions cantherefore be reached regarding efficacy.Theis, 1997 373 Qualitative Some quality issues discussed. Methodological problems mean that available studies provide only very limitedcomparative data on the benefits of different treatment protocolsThomas, 1995 375 Qualitative Comment was made on poor methodological quality of included studies, making definitive conclusions impossibleTowler, 1998 235 Qualitative Design issues discussed narratively. Quality of design generally high. Studies examined according to designvan Balkom, 1997 384 QuantitativeMagnitude of effect size was not significantly associated with quality score.Correlation between quality score and effect Multivariate analysis indicated that eight independent variables including quality score and confounding variablessize accounted for 15% of the variance for panic (F = 0.9; df = 8,42; p = 0.53), and 21% for agoraphobia (F = 2.81;Multiple regression analysis using eight df = 8,87; p = 0.008). In the Discussion section, the authors further state that total quality score was onlyvariables: % of drop-outs, male/female weakly correlated with outcomeratio, duration of illness, age at onset, ageat start of trial, type of diagnosis, % ofpatients with panic disorder alone and thetotal quality scoreVerhagen, 1997 304 Qualitative The methodological quality and results of the studies were discussed separately. Authors conclude that aconclusion regarding efficacy cannot be drawn owing to poor methodological qualityVickers, 1996 201 Qualitative Methodological quality of the studies was discussed in some detail. The authors conclude that the intervention iseffective but acknowledge that methodological flaws may lead some readers to conclude that there is insufficientevidencecontinuedHealth Technology Assessment 2003; Vol. 7: No. 27165

166Review Method of incorporating Results of quality investigationquality into synthesisVickers, 1994 386 Qualitative Some methodological items were discussed in the text and the impact of any limitations on the results of someof the individual studies were discussed. Most of the studies reviewed indicated favourable effects fromhypnotherapy, despite the lack of good control groupsVillalobos, 1998 402 Quantitative Prospective studies found no effect from intervention, whereas retrospective studies (and all studies combined)Sensitivity analyses according to prospective/ suggested that intervention was effectiveretrospective design (prospective studiesmay all be RCTs – not fully clear)Watt, 1998 221 Qualitative All studies supported the use of the intervention in question; results usually statistically or clinically significant, butinconclusive because of small sample size or methodological biasWells-Parker, Quantitative Methodological factors jointly accounted for significant variance (R 2 = 14; p < 0.01). Log (sample size) proved to1995 129 Regression analysis used to investigate be the strongest predictor of ES (β = –0.27; t = 1.29, p = 0.21)impact of methodological factors (including Lower quality studies demonstrated larger ES with larger SDs than high-quality studies [e.g. high-quality (score‘grouping’ quality, source of ES estimate, 5), ES 0.08, SD 0.13; low-quality (score 6), ES 0.25, SD 0.36]. Using 5.0 or 5.5 as the threshold for ‘highpurity or definition of outcome measure quality’ had little impact on the resultsand sample size)The funnel plot indicated that poorer methodological quality was associated with greater variation in effect sizeVarious quality score thresholds were usedas inclusion criteria for the meta-analysisVisual plotWingood, 1996 202 Qualitative Study weaknesses discussed, but more detailed impact of this on results not made. All studies seem to producepositive effects on knowledge and/or behaviour. Authors suggest RCTs more likely to give positive effects, butlittle evidence of this, given only two NRSWitlin, 1998 381 Qualitative Both level I and II evidence supported the use of magnesium sulphate for seizure prophylaxis in women withsevere pre-eclampsiaWright, 1995 203 Qualitative Results of individual studies were discussed with their design and quality. Overall studies were of insufficientquality to allow any strong conclusions to be made. Authors did not state that they used QA, but data extractionsheet covers major quality issues and is very similar to one by Sheldon. May be a modified versionAppendix 7ES, effect size; df, degrees of freedom; RD, risk difference.

Health Technology Assessment 2003; Vol. 7: No. 27Appendix 8Descriptions of the IST and ECSTThe International Stroke Trial(IST) 132Description of the trialThe IST investigated the safety and efficacy ofaspirin and heparin on the outcome of ischaemicstroke. Between January 1991 and May 1996,19,435 patients with suspected acute ischaemicstroke entering 467 hospitals in 36 countries werecentrally randomised using a minimisationalgorithm within 48 h of stroke onset. Using a2 × 2 factorial design, half of the patients wereallocated ‘heparin’ and half ‘avoid heparin’, whilstsimultaneously half were allocated ‘aspirin’ andhalf ‘avoid aspirin’. Aspirin was prescribed at300 mg per day for 14 days (or the duration ofhospital stay). At discharge, clinicians wereencouraged to prescribe all patients long-termaspirin. Among patients allocated heparin,subcutaneous heparin was administered for14 days at one of two randomly allocated doses:5000 IU twice daily (low dose) and 12,500 IUtwice daily (medium dose). All analyses presentedbelow combine these two dose groups.The primary outcomes were death within 14 daysand death or dependency at 6 months. Outcomedata were 99.99% complete for 14-day outcomeand 99.2% complete for 6-month outcome. Interms of compliance, low-dose heparin wasreceived throughout the scheduled treatment by90% and medium-dose heparin by 88% of thoseallocated to it; 94% of those allocated to ‘avoidheparin’ did not receive it. Aspirin was takenthroughout the scheduled treatment period by92% allocated to receive it; 93% of those allocatedto avoid it did not receive it.In both aspirin- and heparin-allocated patientsthere were non-significantly fewer deaths within14 days (9.0% heparin versus 9.3% no heparin;9.0% aspirin versus 9.4% no aspirin). At 6 monthsthere was a non-significant trend towards a smallerpercentage of aspirin group being dead ordependent [62 versus 63%, 2p = 0.07; a differenceof 13 (SD 7) per 1000]. After adjustment forbaseline stroke severity, the benefit from aspirinwas significant [14 events prevented per 1000patients (SD 6), 2p = 0.03]. There was nointeraction between aspirin and heparin in themain outcomes.Baseline characteristics in the ISTThe distribution of 15 baseline characteristicsassociated with prognosis following stroke isreported in Table 36. These data were collectedduring the randomisation process and aretherefore available for every randomised patient.The ORs describe the relationship between achange in the baseline variable and the outcomeof death or disability at 6 months.The trial protocol required the use of a computedtomography (CT) scan to rule out intracranialhaemorrhage. However, where obtaining a CTscan was likely to require a long delay, and theclinician regarded it as very likely that the strokewas ischaemic, a non-comatose patient could berandomised before the CT scan. Table 36 showsthat a high proportion of patients did not reportCT scan results at baseline, and that the absenceof a CT scan indicated poor prognosis. Thisobservation arises through there being arelationship between stroke severity and thedecision to obtain a CT scan.For the purpose of our analyses, the last sixvariables were converted to a score (0–6) giving thenumber of presenting neurological characteristics.The distribution of this score is given in Table 37.Deficit score (Table 37), stroke type andconsciousness (Table 36) appear to have the largestranges of event ranges across their categorisations,indicating that they are likely to be three of themost prognostic variables.Adaptations made to the IST for theresampling projectFor the resampling project, only the comparisonof ‘aspirin’ against ‘avoid aspirin’ was consideredfor the outcome of death or disability at 6 months.To use the IST dataset for this project, themulticentre nature of the trial was exploited toconstruct a series of smaller randomised and nonrandomised‘sub-studies’. Centres falling withingeographical regions were grouped together suchthat sufficient participants were accrued in each167© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Appendix 8TABLE 36 Dead or dependent at 6 months according to treatment and baseline characteristics (IST)Aspirin (n = 9639) No aspirin (n = 9646) OR (95% CI)No. (% adverse outcomes) No. (% adverse outcomes)168Events at 6 months 6000 (62.2) 6125 (63.5) 0.95 (0.89 to 1.01)Delay (h from symptoms)0–3 414 (70.5) 422 (70.6)4–6 1157 (66.1) 1148 (69.5)7–12 2005 (63.5) 2079 (63.3)13–24 2798 (60.8) 2730 (63.0)25–48 3265 (60.3) 3267 (61.0) 0.90 (0.84 to 0.96) per daySexF 4530 (69.5) 4436 (70.5) 1.00M 5109 (55.8) 5210 (57.4) 0.56 (0.53 to 0.60)Age (years)

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 37 Dead or dependent at 6 months according to neurological deficit score (IST)Aspirin (n = 7749) No aspirin (n = 7753) OR (95% CI)Deficit score No. (% adverse outcomes) No. (% adverse outcomes)0 267 (43.8) 246 (39.8)1 719 (41.6) 767 (38.5)2 1373 (53.0) 1421 (51.4)3 2298 (63.4) 2252 (63.4)4 2028 (78.8) 2020 (77.7)5 861 (88.6) 843 (88.6) 1.64 (1.60 to 1.68) per unit increase in score6 203 (92.1) 204 (86.7)TABLE 38 Distribution of baseline characteristics by country (IST)Sample sizeAdverse outcome (%)Delay (h) (mean)Female (%)Age in (years) (mean)Systolic blood pressure (mean)Symptoms on waking (%)Drowsy or unconscious (%)Atrial fibrillation (%)Deficit score (mean)No CT scan (%)Partial anterior +Total anterior strokes (%)Australia 597 56 19 60 71 155 29 26 12 3.0 13 32 + 27 = 59Northern Italy 1911 55 17 52 72 162 31 20 20 3.1 28 39 + 22 = 61Central Italy 980 61 21 60 74 161 24 21 18 2.9 32 50 + 17 = 67Southern Italy 546 60 16 64 70 154 27 20 19 3.2 31 46 + 18 = 64The Netherlands 728 54 16 52 69 169 30 20 10 3.1 14 37 + 27 = 64New Zealand 453 51 25 51 72 155 28 19 17 3.0 4 37 + 22 = 59Norway 526 60 24 53 72 165 28 19 6 2.9 6 44 + 19 = 63Poland 759 53 18 51 69 163 34 28 25 3.3 49 44 + 19 = 63Spain 478 46 19 56 71 157 31 15 15 3.1 21 38 + 24 = 62Sweden 636 42 24 55 75 167 30 15 15 2.5 2 50 + 16 = 66Switzerland 1631 67 17 52 74 164 29 31 16 3.1 26 39 + 28 = 67Scotland 1043 75 23 50 72 158 27 25 21 3.2 46 39 + 29 = 68Northern England and Wales 2762 79 20 53 72 159 27 25 18 3.1 61 41 + 27 = 68Southern England 2452 81 22 51 76 159 27 28 24 3.3 61 40 + 30 = 70arm for the resampling to be undertaken. Inaddition, the sequential nature of the studyrecruitment was used to split participants into‘early recruits’ and ‘later recruits’ to enablehistorically controlled studies to be constructed.Early recruits were those recruited up to andincluding the 15 January 1995; late recruits werethose recruited after this date. This date waschosen to maximise the number of regionsavailable for inclusion in the study and is close tothe median recruitment date.To be considered as a sub-study for the concurrentcohort design, we required that there were at least100 participants within each arm of each trial. Forthe historical cohort design we required that whenthe arms were divided into early and late recruits,there were at least 100 early control recruits and100 late treatment recruits.Fourteen geographical regions were constructedon this basis for the main analysis for bothconcurrent cohort and historical cohort designs.169© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Appendix 8TABLE 39 Distribution of baseline characteristics by country and recruitment period (IST)Before (B) or after (A)Sample sizeAdverse outcome (%)Delay (h) (mean)Female (%)Age (years) (mean)Systolic blood pressure (mean)Symptoms on waking (%)Drowsy or unconscious (%)Atrial fibrillation (%)Deficit score (mean)No CT scan (%)Partial anterior +total anterior strokes (%)Australia B 331 55 20 60 71 155 29 28 14 2.9 12 28 + 27 = 54A 266 57 18 61 72 157 29 24 9 3.0 15 37 + 28 = 65Northern Italy B 1216 53 18 54 72 162 30 20 20 3.1 26 38 + 22 = 61A 695 57 17 49 73 162 32 18 19 3.1 31 41 + 21 = 62Central Italy B 710 60 21 60 74 160 24 20 19 2.9 33 50 + 17 = 66A 270 63 19 58 74 163 26 24 16 3.0 31 50 + 19 = 69Southern Italy B 241 56 16 62 70 155 22 20 18 3.0 29 46 + 15 = 62A 305 63 16 66 70 154 31 20 20 3.3 32 45 + 21 = 66The Netherlands B 521 55 17 53 69 168 30 21 10 3.2 15 36 + 29 = 65A 207 52 16 51 70 170 29 17 11 3.0 12 41 + 23 = 64New Zealand B 250 51 25 49 71 155 29 20 18 3.1 3 36 + 24 = 60A 203 51 26 53 73 156 27 18 15 2.9 4 39 + 19 = 58Norway B 316 65 24 52 73 165 28 21 7 2.8 9 43 + 18 = 62A 210 53 23 56 72 164 27 17 5 3.0 3 44 + 21 = 65Poland B 209 55 18 48 70 160 30 30 27 3.4 33 46 + 22 = 68A 550 53 19 52 69 164 36 28 24 3.2 55 43 + 18 = 61Spain B 237 45 19 58 71 155 32 16 14 2.9 21 40 + 23 = 63A 241 47 19 54 72 159 30 14 16 3.2 21 36 + 26 = 62Sweden B 409 46 24 56 75 166 30 16 16 2.5 1 48 + 17 = 66A 227 33 24 53 75 167 31 14 14 2.5 4 54 + 14 = 68Switzerland B 841 66 16 53 73 162 26 32 17 3.1 28 40 + 29 = 69A 790 68 18 50 75 166 33 30 15 3.1 24 39 + 27 = 66Scotland B 694 73 22 51 71 158 28 26 20 3.2 48 35 + 31 = 66A 349 78 25 49 74 158 26 24 23 3.1 40 46 + 25 = 71Northern England B 1562 80 21 52 72 158 28 25 19 3.2 58 41 + 28 = 69and Wales A 1200 78 19 53 72 160 25 25 17 3.0 65 42 + 26 = 68Southern England B 1384 82 21 51 76 159 25 28 23 3.3 62 40 + 30 = 70A 1068 80 24 50 76 157 29 28 24 3.3 60 41 + 30 = 71170

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 40 Distribution of baseline characteristics in 10 UK cities combining results across 28 hospitals (IST)Sample sizeAdverse outcome (%)Delay (h) (mean)Female (%)Age (years) (mean)Systolic blood pressure (mean)Symptoms on waking (%)Drowsy or unconscious (%)Atrial fibrillation (%)Deficit score (mean)No CT scan (%)Partial anterior +total anterior strokes (%)Belfast 234 85 26 47 76 151 30 38 22 3.1 28 40 + 29 = 70Bishop Auckland 234 80 16 48 74 159 30 31 19 3.2 75 41 + 31 = 72Blackburn 221 83 15 52 72 161 30 21 18 3.2 74 45 + 24 = 69Edinburgh 329 57 22 49 69 159 29 20 20 3.0 34 36 + 27 = 63Leeds 226 58 19 54 65 160 28 19 10 2.9 64 34 + 24 = 58Liverpool 536 80 17 54 72 162 26 19 19 2.8 89 49 + 21 = 71London 344 82 24 49 76 160 27 26 23 3.2 12 39 + 27 = 66Newcastle 361 83 26 46 75 156 26 31 21 3.3 25 39 + 32 = 71Nottingham 305 83 17 50 77 161 22 24 22 3.2 97 31 + 30 = 61Sheffield 308 71 21 58 71 159 27 21 14 3.0 50 38 + 23 = 61The distribution of the outcome of dead ordependent at 6 months and of the baselinecharacteristics in these sub-studies are described inTable 38. Table 39 describes the baselinecharacteristics for the cohorts split according to‘early recruits’ (before) and ‘late recruits’ (after).Differences are evident in both tables in thefrequency of the outcome and for some of thebaseline characteristics, notably the percentagewith total anterior strokes, or admitted ‘drowsy orunconscious’.These 14 regions contain data from 15,502 (80%)of the randomised participants. The 3933excluded participants were recruited in Argentina,Austria, Belgium, Brazil, Canada, Chile, CzechRepublic, Denmark, Eire, Finland, France, Greece,Hong Kong, Hungary, India, Israel, Japan,Portugal, Romania, Singapore, Slovak Republic,Slovenia, South Africa, Sri Lanka, Turkey and theUSA, but without sufficient numbers in any regionto construct an additional centre.A second set of 10 sub-studies was constructedfrom the UK data, by grouping hospitals withincities, including a total of 3098 participants (50%of those recruited in the UK). The distribution ofbaseline variables across the 10 cities is shown inTable 40. Data from these sub-studies were usedonly to construct concurrently controlled studies,insufficient data being available for the historicallycontrolled comparisons.The European Carotid SurgeryTrial (ECST) 133Description of the trialThe ECST investigated the risks and benefits ofcarotid endarterectomy, primarily in terms ofstroke prevention. Between October 1981 andMarch 1994, 3024 patients with recentlysymptomatic carotid stenosis entering 100centres in 14 countries were randomised tocarotid endarterectomy or control (avoid surgeryas long as possible). Patients were eligible forinclusion if they had had at least one transient ormild symptomatic ischaemic vascular event withinthe last 6 months, and had some degree ofcarotid stenosis. Randomisation was performedcentrally using a minimisation algorithm, 60% ofpatients being allocated to surgery and 40% tocontrol.Of the 1811 allocated surgery, 1745 (96%) hadreceived it within 1 year. Of the 1213 allocatedcontrol, 42 (3%) received surgery within 1 year.The control treatment usually consisted of advice171© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Appendix 8TABLE 41 Baseline characteristics according to country (ECST)Sample sizeAdverse outcome (%)Female (%)Age (years) (mean)Previous major stroke (%)Neurological signs (%)Previous MI (%)Angina (%)Prophylactic aspirin use (%)>50% stenosis (%)Region 1 458 31 33 62 75 73 86 79 45 70Region 2 271 43 22 66 65 69 92 85 75 48Region 3 294 39 26 64 70 63 89 86 35 68Region 4 264 27 21 63 67 65 91 92 45 50Region 5 458 30 27 60 71 71 90 87 54 58Region 6 308 46 35 61 79 74 82 77 14 81Region 7 439 36 33 61 80 79 88 80 33 73Region 8 341 51 28 62 77 72 86 79 39 73MI, myocardial infarction.TABLE 42 Baseline characteristics according to country and recruitment period (ECST)Before (B) or after (A)Sample sizeAdverse outcome (%)Female (%)Age (years) (mean)Previous major stroke (%)Neurological signs (%)Previous MI (%)Angina (%)Prophylactic aspirin use (%)>50% stenosis (%)Region 1 B 131 45 36 60 75 31 10 15 41 54A 327 26 32 63 75 26 16 23 61 76Region 2 B 149 54 23 66 69 32 7 15 21 39A 122 30 21 67 61 30 10 15 29 60Region 3 B 119 49 29 63 76 34 10 8 71 60A 175 33 24 64 66 38 12 17 60 74Region 4 B 128 33 21 61 66 38 11 4 57 43A 136 22 21 64 68 32 7 13 54 57Region 5 B 243 39 26 59 70 30 13 13 38 48A 215 21 28 61 73 27 7 14 55 69Region 6 B 136 59 35 60 80 28 17 23 82 79A 172 37 35 62 77 24 18 23 90 82Region 7 B 201 55 35 61 81 18 14 23 56 70A 143 32 31 61 79 24 10 16 83 78172Region 8 B 231 53 25 62 73 34 13 22 48 73A 205 24 32 63 81 21 16 20 77 72

Health Technology Assessment 2003; Vol. 7: No. 27against smoking, treatment of raised bloodpressure and antiplatelet drugs, although itwas noted to vary between centres and over theyears.The primary outcome was death or major stroke.Overall there was a non-significant difference inoutcome (37.0% of surgery-group patients versus36.5% of control-group patients), although therewas a relationship between benefit of surgery anddegree of stenosis, surgery being of value when thedegree of stenosis was >80%.Baseline characteristics in the ECSTEight variables for baseline characteristicsassociated with prognosis were included in thedata set. Details of the values of these variables arenot given in the report as they will soon bepublished in other research publications producedby the trial collaborative group. Thesecharacteristics were collected during therandomisation process, and were thereforeavailable for every randomised patient.Adaptations made to the ECST for theresampling projectTo use the ECST dataset for this project, themulticentre nature of the trial was exploited toconstruct a series of smaller randomised and nonrandomisedstudies, in the same way as for theIST. Centres falling within geographical regionswere grouped together such that sufficientparticipants were accrued in each arm for theresampling to be undertaken. In addition, thesequential nature of the study recruitment wasused to split participants into ‘early recruits’ and‘later recruits’ to enable historically controlledstudies to be constructed. For all regions otherthan Region 4, early recruits were those recruitedup to and including 5 December 1987; laterecruits were those recruited after this date. A cutoffdate of 19 December 1988 was used for Region4. These dates were chosen to maximise thenumber of regions available for inclusion in thestudy.Sample sizes used in the resampling for the ECSTstudy were smaller than those for the IST. To beconsidered as a sub-study for the concurrentcohort design, we required that there were at least40 participants within each arm of each trial. Forthe historical cohort design we required that whenthe arms were divided into early and late recruitsthere were at least 40 early control recruits and 40late treatment recruits.Eight geographical regions were constructed onthis basis for the main analysis for both concurrentcohort and historical cohort designs. As centreswere smaller than for the IST, neighbouringcountries were grouped where necessary. Theseeight regions recruited 2833 (94%) of trialparticipants. For reasons of future publications,the regions are not identifiable in the tables wepresent.The ECST is one of two trials investigating theefficacy and safety of carotid endarterectomy.Both the ECST and the second trial, NASCET, 135published early results during the recruitmentperiod of the ECST, which led to the datamonitoring committee of the ECST changingthe inclusion criteria of the ECST in 1990.Cases recruited after this change were deletedfrom the data set where this change would beconfounded with the generation of groups(e.g. for the historically controlled studyanalyses).The distribution of the outcome of dead or majorstroke, and of the baseline characteristics in thesesub-studies are described in Table 41. Table 42describes the baseline characteristics for thecohorts split according to ‘early recruits’ (before)and ‘late recruits’ (after). Differences are evidentin both tables in the frequency of the outcome andfor some of the baseline characteristics, but aregreatest in the historical comparisons. In thesecentres the event rates are lower in the secondperiod in all eight centres, whereas thepercentages with high degrees of stenosis increasewith time in all but one centre. The patternobserved with degree of stenosis is likely to reflecta change in case-mix due to emerging evidence ofthe dangers of surgery for low-grade surgeryduring the trial’s recruitment period.173© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Health Technology Assessment 2003; Vol. 7: No. 27Health Technology AssessmentProgrammeMembersPrioritisation Strategy GroupChair,Professor Kent Woods,Director, NHS HTA Programme& Professor of Therapeutics,University of LeicesterProfessor Bruce Campbell,Consultant Vascular & GeneralSurgeon, Royal Devon & ExeterHospitalProfessor Shah Ebrahim,Professor in Epidemiologyof Ageing, University ofBristolDr John Reynolds, ClinicalDirector, Acute GeneralMedicine SDU, RadcliffeHospital, OxfordDr Ron Zimmern, Director,Public Health Genetics Unit,Strangeways ResearchLaboratories, CambridgeMembersHTA Commissioning BoardProgramme Director,Professor Kent Woods, Director,NHS HTA Programme,Department of Medicine andTherapeutics, Leicester RoyalInfirmary, Robert KilpatrickClinical Sciences Building,LeicesterChair,Professor Shah Ebrahim,Professor in Epidemiology ofAgeing, Department of SocialMedicine, University of Bristol,Canynge Hall, WhiteladiesRoad, BristolDeputy Chair,Professor Jenny Hewison,Professor of Health CarePsychology, Academic Unit ofPsychiatry and BehaviouralSciences, University of LeedsSchool of Medicine, LeedsProfessor Douglas Altman,Professor of Statistics inMedicine, Centre for Statisticsin Medicine, Oxford University,Institute of Health Sciences,Cancer Research UK MedicalStatistics Group, Headington,OxfordProfessor John Bond, Professorof Health Services Research,Centre for Health ServicesResearch, University ofNewcastle, School of HealthSciences, Newcastle upon TyneProfessor John Brazier, Directorof Health Economics, SheffieldHealth Economics Group,School of Health & RelatedResearch, University ofSheffield, ScHARR RegentCourt, SheffieldDr Andrew Briggs, PublicHealth Career Scientist, HealthEconomics Research Centre,University of Oxford, Instituteof Health Sciences, OxfordDr Christine Clark, MedicalWriter & Consultant Pharmacist,Cloudside, Rossendale, LancsandPrincipal Research Fellow,Clinical Therapeutics in theSchool of Pharmacy, BradfordUniversity, BradfordProfessor Nicky Cullum,Director of Centre for EvidenceBased Nursing, Department ofHealth Sciences, University ofYork, Research Section,Seebohm Rowntree Building,Heslington, YorkDr Andrew Farmer, SeniorLecturer in General Practice,Department of Primary HealthCare, University of Oxford,Institute of Health Sciences,Headington, OxfordProfessor Fiona J Gilbert,Professor of Radiology,Department of Radiology,University of Aberdeen, LilianSutton Building, Foresterhill,AberdeenProfessor Adrian Grant,Director, Health ServicesResearch Unit, University ofAberdeen, Drew Kay Wing,Polwarth Building, Foresterhill,AberdeenProfessor Alastair Gray, Director,Health Economics ResearchCentre, University of Oxford,Institute of Health Sciences,Headington, OxfordProfessor Mark Haggard,Director, MRC ESS Team, CBUElsworth House, Addenbrooke’sHospital, CambridgeProfessor F D Richard Hobbs,Professor of Primary Care &General Practice, Department ofPrimary Care & GeneralPractice, University ofBirmingham, Primary Care andClinical Sciences Building,Edgbaston, BirminghamProfessor Peter Jones, Head ofDepartment, UniversityDepartment of Psychiatry,University of Cambridge,Addenbrooke's Hospital,CambridgeProfessor Sallie Lamb, ResearchProfessor in Physiotherapy/Co-Director, InterdisciplinaryResearch Centre in Health,Coventry University, CoventryDr Donna Lamping, SeniorLecturer, Health ServicesResearch Unit, Public Healthand Policy, London School ofHygiene and Tropical Medicine,LondonCurrent and past membership details of all HTA ‘committees’ are available from the HTA website (www.ncchta.org)Professor David Neal, Professorof Surgical Oncology, OncologyCentre, Addenbrooke's Hospital,CambridgeProfessor Tim Peters, Professorof Primary Care Health ServicesResearch, Division of PrimaryHealth Care, University ofBristol, Cotham House, CothamHill, BristolProfessor Ian Roberts, Professorof Epidemiology & PublicHealth, Intervention ResearchUnit, London School ofHygiene and Tropical Medicine,LondonProfessor Peter Sandercock,Professor of Medical Neurology,Department of ClinicalNeurosciences, University ofEdinburgh, Western GeneralHospital NHS Trust, BramwellDott Building, EdinburghProfessor Martin Severs,Professor in Elderly HealthCare, Portsmouth Institute ofMedicine, Health & Social Care,St George’s Building,PortsmouthDr Jonathan Shapiro, SeniorFellow, Health ServicesManagement Centre, ParkHouse, Birmingham183© Queen’s Printer and Controller of HMSO 2003. All rights reserved.

Health Technology Assessment ProgrammeMembersDiagnostic Technologies & Screening PanelChair,Dr Ron Zimmern, Director ofthe Public Health Genetics Unit,Strangeways ResearchLaboratories, CambridgeDr Paul Cockcroft, ConsultantMedical Microbiologist/Laboratory Director, PublicHealth Laboratory,St Mary’s Hospital,PortsmouthProfessor Adrian K Dixon,Professor of Radiology,Addenbrooke’s Hospital,CambridgeDr David Elliman, Consultant inCommunity Child Health,LondonDr Andrew Farmer, SeniorLecturer in General Practice,Institute of Health Sciences,University of OxfordDr Karen N Foster, ClinicalLecturer, Dept of GeneralPractice & Primary Care,University of AberdeenProfessor Jane Franklyn,Professor of Medicine,University of BirminghamProfessor Antony J Franks,Deputy Medical Director, TheLeeds Teaching Hospitals NHSTrustMr Tam Fry, HonoraryChairman, Child GrowthFoundation, LondonDr Susanne M Ludgate, MedicalDirector, Medical DevicesAgency, LondonDr William Rosenberg, SeniorLecturer and Consultant inMedicine, University ofSouthamptonDr Susan Schonfield, CPHMSpecialised ServicesCommissioning, CroydonPrimary Care TrustDr Margaret Somerville,Director of Public Health,Teignbridge Primary Care Trust,DevonMr Tony Tester, Chief Officer,South Bedfordshire CommunityHealth Council, LutonDr Andrew Walker, SeniorLecturer in Health Economics,University of GlasgowProfessor Martin J Whittle,Head of Division ofReproductive & Child Health,University of BirminghamDr Dennis Wright, ConsultantBiochemist & Clinical Director,Pathology & The KennedyGalton Centre, Northwick Park& St Mark’s Hospitals, HarrowMembersPharmaceuticals PanelChair,Dr John Reynolds, ClinicalDirector, Acute GeneralMedicine SDU, OxfordRadcliffe HospitalProfessor Tony Avery, Professorof Primary Health Care,University of NottinghamProfessor Iain T Cameron,Professor of Obstetrics &Gynaecology, University ofSouthamptonMr Peter Cardy, ChiefExecutive, Macmillan CancerRelief, LondonDr Christopher Cates, GP andCochrane Editor, Bushey HealthCentre, Bushey, Herts.Mr Charles Dobson, SpecialProjects Adviser, Department ofHealthDr Robin Ferner, ConsultantPhysician and Director, WestMidlands Centre for AdverseDrug Reactions, City HospitalNHS Trust, BirminghamDr Karen A Fitzgerald,Pharmaceutical Adviser, Bro TafHealth Authority, CardiffProfessor Alastair Gray,Professor of Health Economics,Institute of Health Sciences,University of OxfordMrs Sharon Hart, ManagingEditor, Drug & TherapeuticsBulletin, LondonDr Christine Hine, Consultant inPublic Health Medicine,Bristol South & West PrimaryCare TrustProfessor Robert Peveler,Professor of Liaison Psychiatry,Royal South Hants Hospital,SouthamptonDr Frances Rotblat, CPMPDelegate, Medicines ControlAgency, LondonMrs Katrina Simister, NewProducts Manager, NationalPrescribing Centre, LiverpoolDr Ken Stein, Senior Lecturer inPublic Health, University ofExeterProfessor Terence Stephenson,Professor of Child Health,University of NottinghamDr Richard Tiner, MedicalDirector, Association of theBritish Pharmaceutical Industry,LondonProfessor Dame Jenifer Wilson-Barnett, Head of FlorenceNightingale School of Nursing& Midwifery, King’s College,London184Current and past membership details of all HTA ‘committees’ are available from the HTA website (www.ncchta.org)

Health Technology Assessment 2003; Vol. 7: No. 27MembersTherapeutic Procedures PanelChair,Professor Bruce Campbell,Consultant Vascular andGeneral Surgeon, Royal Devon& Exeter HospitalDr Mahmood Adil, Head ofClinical Support & HealthProtection, Directorate ofHealth and Social Care (North),Department of Health,ManchesterProfessor John Bond, Head ofCentre for Health ServicesResearch, University ofNewcastle upon TyneMr Michael Clancy, Consultantin A & E Medicine,Southampton General HospitalDr Carl E Counsell, SeniorLecturer in Neurology,University of AberdeenDr Keith Dodd, ConsultantPaediatrician, DerbyshireChildren’s Hospital, DerbyProfessor Gene Feder, Professorof Primary Care R&D, Barts &the London, Queen Mary’sSchool of Medicine andDentistry, University of LondonMs Bec Hanley, FreelanceConsumer Advocate,Hurstpierpoint, West SussexProfessor Alan Horwich,Director of Clinical R&D, TheInstitute of Cancer Research,LondonDr Phillip Leech, PrincipalMedical Officer for PrimaryCare, Department of Health,LondonMr George Levvy, ChiefExecutive, Motor NeuroneDisease Association,NorthamptonProfessor James Lindesay,Professor of Psychiatry for theElderly, University of LeicesterDr Mike McGovern, SeniorMedical Officer, Heart Team,Department of Health, LondonDr John C Pounsford,Consultant Physician, NorthBristol NHS TrustProfessor Mark Sculpher,Professor of Health Economics,Institute for Research in theSocial Services, University ofYorkDr L David Smith, ConsultantCardiologist, Royal Devon &Exeter HospitalProfessor Norman Waugh,Professor of Public Health,University of AberdeenCurrent and past membership details of all HTA ‘committees’ are available from the HTA website (www.ncchta.org)185

Health Technology Assessment ProgrammeMembersExpert Advisory NetworkMr Gordon Aylward,Chief Executive,Association of British Health-Care Industries, LondonMs Judith Brodie,Head of Cancer SupportService, Cancer BACUP, LondonMr Shaun Brogan,Chief Executive, RidgewayPrimary Care Group, Aylesbury,BucksMs Tracy Bury,Project Manager, WorldConfederation for PhysicalTherapy, LondonMr John A Cairns,Professor of Health Economics,Health Economics ResearchUnit, University of AberdeenProfessor Howard Stephen Cuckle,Professor of ReproductiveEpidemiology, Department ofPaediatrics, Obstetrics &Gynaecology, University ofLeedsProfessor Nicky Cullum,Director of Centre for EvidenceBased Nursing, University of YorkDr Katherine Darton,Information Unit, MIND – TheMental Health Charity, LondonProfessor Carol Dezateux,Professor of PaediatricEpidemiology, LondonProfessor Martin Eccles,Professor of ClinicalEffectiveness, Centre for HealthServices Research, University ofNewcastle upon TyneProfessor Pam Enderby,Professor of CommunityRehabilitation, Institute ofGeneral Practice and PrimaryCare, University of SheffieldMr Leonard R Fenwick,Chief Executive, Newcastleupon Tyne Hospitals NHS TrustProfessor David Field,Professor of Neonatal Medicine,Child Health, The LeicesterRoyal Infirmary NHS TrustMrs Gillian Fletcher,Antenatal Teacher & Tutor andPresident, National ChildbirthTrust, Henfield, West SussexMs Grace Gibbs,Deputy Chief Executive,Director for Nursing, Midwifery& Clinical Support Servs., WestMiddlesex University Hospital,Isleworth, MiddlesexDr Neville Goodman,Consultant Anaesthetist,Southmead Hospital, BristolProfessor Robert E Hawkins,CRC Professor and Director ofMedical Oncology, Christie CRCResearch Centre, ChristieHospital NHS Trust, ManchesterProfessor F D Richard Hobbs,Professor of Primary Care &General Practice, Department ofPrimary Care & GeneralPractice, University ofBirminghamProfessor Allen Hutchinson,Director of Public Health &Deputy Dean of ScHARR,Department of Public Health,University of SheffieldProfessor Rajan Madhok,Medical Director & Director ofPublic Health, Directorate ofClinical Strategy & PublicHealth, North & East Yorkshire& Northern Lincolnshire HealthAuthority, YorkProfessor David Mant,Professor of General Practice,Department of Primary Care,University of OxfordProfessor Alexander Markham,Director, Molecular MedicineUnit, St James’s UniversityHospital, LeedsDr Chris McCall,General Practitioner, TheHadleigh Practice, CastleMullen, DorsetProfessor Alistair McGuire,Professor of Health Economics,London School of EconomicsDr Peter Moore,Freelance Science Writer,Ashtead, SurreyDr Andrew Mortimore,Consultant in Public HealthMedicine, Southampton CityPrimary Care TrustDr Sue Moss,Associate Director, CancerScreening Evaluation Unit,Institute of Cancer Research,Sutton, SurreyProfessor Jon Nicholl,Director of Medical CareResearch Unit, School of Healthand Related Research,University of SheffieldMrs Julietta Patnick,National Co-ordinator, NHSCancer Screening Programmes,SheffieldProfessor Chris Price,Visiting Chair – Oxford, ClinicalResearch, Bayer DiagnosticsEurope, CirencesterMs Marianne Rigge,Director, College of Health,LondonProfessor Sarah Stewart-Brown,Director HSRU/HonoraryConsultant in PH Medicine,Department of Public Health,University of OxfordProfessor Ala Szczepura,Professor of Health ServiceResearch, Centre for HealthServices Studies, University ofWarwickDr Ross Taylor,Senior Lecturer, Department ofGeneral Practice and PrimaryCare, University of AberdeenMrs Joan Webster,Consumer member, HTA –Expert Advisory Network186Current and past membership details of all HTA ‘committees’ are available from the HTA website (www.ncchta.org)

FeedbackThe HTA Programme and the authors would like to knowyour views about this report.The Correspondence Page on the HTA website(http://www.ncchta.org) is a convenient way to publishyour comments. If you prefer, you can send your commentsto the address below, telling us whether you would likeus to transfer them to the website.We look forward to hearing from you.The National Coordinating Centre for Health Technology Assessment,Mailpoint 728, Boldrewood,University of Southampton,Southampton, SO16 7PX, UK.Fax: +44 (0) 23 8059 5639 Email: hta@soton.ac.ukhttp://www.ncchta.org ISSN 1366-5278