Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27significance compared with only 9% of comparableRCTs. Taking a significant result as leading to aconclusion of effectiveness, selection bias in theseconcurrently controlled studies was strong enoughfor 41% to reach unjustified conclusions. Notably,21% of these studies wrongly concluded thataspirin was harmful. Similarly, 40% of the ECSThistorically controlled studies were statisticallysignificant compared with 15% of RCTs, implyingthat the use of historical comparisons introducedbias severe enough to lead to unjustifiedconclusions in favour of surgery in 25% of thestudies. This high frequency of ‘conclusionchanging’bias raises concern.However, are these frequencies and magnitudes ofbias likely to be typical? In many clinical situationsthe variation in case-mix is large, as, for example,in the treatment of those who have suffered astroke, who, as seen in the IST trial, vary in theirlevels of consciousness, paralysis and many otherprognostic factors. Here the possibility ofproducing groups with large differences in casemix(resulting in serious bias) is high, such thatthe results of non-randomised comparisons areunlikely to be reliable. In other situations there islittle prognostic information available at the timeof treatment allocation, so that the possibility ofintroducing case-mix related biases might belower. For example, in childhood vaccination trialsin the developed world there is little possibility ofpredicting the occurrence of infectious disease atthe time of vaccine administration. It couldtherefore be hypothesised that non-randomisedstudies in such contexts would be less likely to bebiased. These issues require further research.Can we predict bias?One issue that is clearly important when trying topredict bias is the quality of the study design.Whilst there is hardly any empirical evidenceabout the importance of particular dimensions ofstudy quality for non-randomised studies, theimportance of evaluating the quality of RCTs forthe likelihood of performance, attrition anddetection bias has been well established, and thereis good empirical evidence of bias associated withcertain methodological features. 23 Despite the lackof empirical evidence specific to non-randomisedstudies, it is likely that they are equally if not moresusceptible to these biases than RCTs.It has been suggested that to assess the likelihoodof selection bias in a non-randomised study, it isnecessary to know the methods by whichparticipants were allocated to interventiongroups. 18 Specifically, in order to assess the degreeof selection bias we need to know the factorsinfluencing the allocation, the manner in whichthose factors are related to outcome and how thefactors were dealt with in the analysis. Theserequirements underlie our choice of the fourcore items we propose for inclusion in qualityassessment tools for non-randomised studies(Chapter 4), although we found that theywere rarely included in the tools that weassessed.In classical non-randomised designs, allocation isoften made according to centre (concurrentcontrolled cohort studies) or period (historicallycontrolled cohort studies). However, knowing thatthere is a difference in location or era of thetreated and control samples is inadequate to judgecomparability: it is necessary to know the mannerin which these differences lead to specificdifferences in case-mix.The situation is more complex when interventionsbeing compared have been used across centresduring the same period, where allocation is byindication or patient preference. Then it is notusually possible to describe fully the factorsinfluencing allocation, nor is it likely that theallocation mechanism will be consistent acrosspatients and clinicians. In addition, patients maybe included in a study who are suitable for onlyone of the two treatments. These factors all meanthat allocation may be strongly linked toprognostic factors, such that the bias related todifferences in case-mix will be large. 3Theoretically, the second stage in assessing thelikelihood of bias in a non-randomised study is toconsider whether differences in confoundingfactors are comparable between the groups and, ifnot, whether they have been adjusted for in theanalysis. Many investigators routinely claim effectsas ‘adjusted’ or ‘independent’ or ‘unbiased’ whenthey have used statistical methods to correct fordifferences in case-mix. However, their trust in theobserved comparability of case-mix and the abilityof statistical adjustment methods to adjustproperly for differences in case-mix is unfounded.The unique design of our investigation hasallowed the performance of these case-mixadjustment methods to be assessed by comparingunadjusted and adjusted results with those fromrandomised comparisons. We have onlyundertaken these evaluations in two data sets andwe found consistent results indicating that casemixadjustment cannot reliably compensate forbiases in selection, and in fact it may introduceadditional bias.89© Queen’s Printer and Controller of HMSO 2003. All rights reserved.