Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27TABLE 25 Comparison of methods of case-mix adjustment applied to results of historically controlled studies with results of RCTsresampled from 14 regions within the ISTPercentage of studies withstatistically significantAverageVariability of results results (p < 0.05)OR SD of log OR Ratio with RCT Benefit Harm TotalRCTs 0.89 0.35 9 2 11Historically controlled studiesUnadjusted 0.88 0.44 1.23 16 4 20Stratification 0.88 0.53 1.51 17 5 22Logistic regressionFull model a 0.85 0.56 1.60 13 3 16Stepwise p r = 0.05 b 0.84 0.52 1.49 15 3 18Stepwise p r = 0.15 c 0.85 0.53 1.51 14 3 17Propensity scoreMatched d 0.91 0.43 1.23 7 2 9Stratified 0.90 0.40 1.14 9 2 11Regression 0.91 0.39 1.11 9 2 11a Full model includes 10 covariates.b Mean number of covariates included: 4.5.c Mean number of covariates included: 5.7.d Mean number of patients matched: 132 out of 200.considered by region in the last six columns ofTable 26. Stratification reduced systematic bias inonly two of the 14 regions, logistic regressionadjustments reduced bias in six regions,propensity score methods reduced bias in five ofthese six, and in two additional regions. Overall,bias introduced by use of an historical controlgroup was consistently reduced by case-mixadjustment methods in less than half of the regions.Where biases were increased by adjustment, thedirection of the increase was unpredictable. InScotland the historical control result (OR = 1.23)suggested the treatment to be harmful, in contrastto a beneficial result observed in the RCTs(OR = 0.78). Logistic regression further increasedthis bias (OR = 1.40). In Norway the oppositepattern was seen, with adjustment by logisticregression (OR = 0.41) increasing theoverestimate of treatment benefit in the historicalcontrols (OR = 0.48) compared with RCTs(OR = 0.78). However, in some regions, such asThe Netherlands, adjustment moved the historicalcontrol estimates (OR = 0.85) from a value whichwas lower than the RCTs (OR = 0.88) to a highervalue (OR = 1.14), changing a relatively correctestimate of the benefit of the intervention to abiased estimate suggestive of harm. Similar, butless extreme, changes occurred with propensityscore methods.Unpredictability in bias originating fromconcurrently controlled studiesUnpredictability in bias was observed most clearlyin the IST concurrently controlled comparisons.The ability of the seven case-mix adjustmentmethods to correct these biases is summarised inTables 27 and 28 for regional comparisons in theIST and UK city comparisons in the IST. Regionalcomparisons in the ECST are given in Table 29 forcompleteness. The results for the studiesdemonstrating the largest unpredictable biases,the regional IST comparison (see Chapter 6), arealso shown in Figure 16.As with the historically controlled studies, logisticregression increased the variability of results for allthree situations, the increased spread of resultsbeing evident in Figure 16. Use of the full logisticmodel (including all covariates) increased spreadmore than use of stepwise models. Again, therewas little corresponding increase in spurioussignificance levels as the power of the analyses wasreduced. Propensity score methods slightlyreduced unpredictable bias and spurioussignificance rates for two of the three situations,while stratification made little difference.Although there was no evidence of a systematicbias in the unadjusted results, both logisticregression and propensity score methodsintroduced small systematic biases in most73© Queen’s Printer and Controller of HMSO 2003. All rights reserved.