Evaluating non-randomised intervention studies - NIHR Health ...

Executive summaryUse of quality assessment in systematicreviews of non-randomisedstudiesOf 511 systematic reviews that included nonrandomisedstudies, only 169 (33%) assessed studyquality. Many used quality assessment toolsdesigned for RCTs or developed by the authorsthemselves, and did not include key qualitycriteria relevant to non-randomised studies. Sixtyninereviews investigated the impact of quality onstudy results in a quantitative manner.Empirical estimates of bias associatedwith non-random allocationThe bias introduced by non-random allocation wasnoted to have two components. First, the biascould lead to consistent over- or underestimationsof treatment effects. This occurred for historicalcontrols, the direction of bias depending on timetrends in the case-mix of participants recruited tothe study. Second, the bias increased variation inresults for both historical and concurrent controls,owing to haphazard differences in case-mixbetween groups. The biases were large enough tolead studies falsely to conclude significant findingsof benefit or harm.Empirical evaluation of case-mixadjustment methodsFour strategies for case-mix adjustment wereevaluated: none adequately adjusted for bias inhistorically and concurrently controlled studies.Logistic regression on average increased bias.Propensity score methods performed better, butwere not satisfactory in most situations. Detailedinvestigation revealed that adequate adjustmentcan only be achieved in the unrealistic situationwhen selection depends on a single factor.Omission of important confounding factors canexplain underadjustment. Correlatedmisclassifications and measurement error inconfounding variables may explain the observedincrease in bias with logistic regression, as maydifferences between conditional and unconditionalodds ratio estimates of treatment effects.ConclusionsResults of non-randomised studies sometimes, butnot always, differ from results of randomisedstudies of the same intervention. Non-randomisedstudies may still give seriously misleading resultswhen treated and control groups appear similar inkey prognostic factors. Standard methods of casemixadjustment do not guarantee removal of bias.Residual confounding may be high even whengood prognostic data are available, and in somesituations adjusted results may appear more biasedthan unadjusted results.Although many quality assessment tools exist andhave been used for appraising non-randomisedstudies, most omit key quality domains. Six toolswere considered potentially suitable for use insystematic reviews, but each requires revision tocover all relevant quality domains.Healthcare policies based upon non-randomisedstudies or systematic reviews of non-randomisedstudies may need re-evaluation if the uncertaintyin the true evidence base was not fully appreciatedwhen policies were made.The inability of case-mix adjustment methods tocompensate for selection bias and our inability toidentify non-randomised studies which are free ofselection bias indicate that non-randomisedstudies should only be undertaken when RCTs areinfeasible or unethical.Recommendations for furtherresearch●●●●●The resampling methodology utilised hereshould be applied in other clinical areas toascertain whether the biases we describe aretypical.Efforts should be focused on developing qualityassessment tools for non-randomised studies,possibly by refining existing tools.Research should consider how qualityassessment of non-randomised studies can beincorporated into reviews, and the implicationsof individual quality features for interpretationof a review’s results.Reasons for the apparent failure of case-mixadjustment methods should be furtherinvestigated, including assessments of thegeneralisability of our results to risk assessmentsand epidemiological studies, and differencesbetween conditional and unconditionalestimates of treatment effects.The role of the propensity score should befurther evaluated, and computer macros madeavailable for its application.x