Evaluating non-randomised intervention studies - NIHR Health ...

Health Technology Assessment 2003; Vol. 7: No. 27was made to investigate interactions, or toinvestigate non-linear models of continuouscovariates. Although these could be regarded asimportant oversimplifications, the size of the dataset (n = 200 for IST, n = 80 for ECST) means thatsuch investigations would be underpowered.Propensity score methodsAs noted above, an individual’s propensity score istheir probability of being in the treatment groupgiven their baseline data. Once the score has beencalculated for all individuals in the data set, it canbe used in the analysis in one of three ways. First,participants in the experimental and controlgroups can be selected to generate groups withsimilar distributions of propensity score bymatching each individual in the experimentalgroup with an individual in the control group whoshares the same (or a very similar) propensityscore. Participants for whom no match can befound are excluded from the analysis. Second, astratified analysis can be undertaken, theparticipants being divided into, say, quintilesaccording to their propensity score, andcomparisons made within each of the five groups.As with conventional stratified analysis, the overalltreatment effect is estimated by calculating aweighted average of the within strata estimates ofthe treatment effect. Third, the propensity scorecan be used as a predictor of outcome in a logisticregression model. The model will estimate therelationship between propensity score andoutcome, and make a suitable adjustment to theestimate of treatment effect according to thedifference in mean propensity score betweentreated and control groups.The first stage of the propensity score methodinvolved calculating propensity probabilities for allmembers of each data set, using a logisticregression model with ‘treatment’ as the outcomevariable. The standard sets of 10 covariates for theIST and eight covariates for the ECST (as listed inTable 20) were included in this model, with nointeraction terms being considered. Thelogistic command in STATA was used to fit themodel and obtain estimated propensity scores foreach participant. The propensity score was thenused in the three different ways: matching,stratification and regression.Matching on propensity scores involved pairingeach treated participant with a control participantwho has an identical or very similar propensityscore. Our definition of ‘very similar’ wasRosenbaum and Rubin’s suggestion of beingwithin 0.25 SD of the logit of the propensityscore. 145 This approach takes into account theobserved variability of the computed propensityscores in each study. The difference d in logitpropensity scores was calculated for all possiblepairs of treated and control participants. Thepairings were sorted in ascending order accordingto d, and pairs where d was greater than ourmatching criterion were dropped. Then the firstmatched pair (t 1 , c 1 ) was selected, and all otherpairs including either of t 1 and c 1 dropped. Theprocess was repeated through the data set,selecting the next available matched pair (t i , c i ),and dropping all other pairs including either of t ior c i . Participants in both groups who were notmatched were excluded from the analysis. Theindividual pairing was not exploited in theanalysis: the OR was computed as the ratio of theodds of the outcome in the treated group to theodds in the control group in the standard way,with a standard error for log OR calculated usingthe large sample estimate.The stratified analysis used the Mantel–Haenszelmethod to estimate a common OR across theparticipants grouped according to their propensityscore. In each analysis five groups were used,defined by calculating the quintiles of theobserved propensity scores (pooled acrosstreatment and control groups). Calculations wereperformed using the STATA mhodds command.The regression analysis estimated the treatmenteffect adjusted for differences in propensity scoreby fitting a logistic regression model to theoutcome data with just two covariates, thetreatment group and the log OR of the propensityscore. The model was fitted using the STATAlogistic command.Statistical analysesAs with the analysis of bias related to study designs(Chapter 6), the focus of the statistical analysis wason describing and comparing the location(average) and spread (variability) of the treatmenteffects. Distributions were considered first for theRCTs and the non-randomised studies withoutadjustment, and second for the non-randomisedstudies with each of the eight case-mix strategiesdescribed above. In all analyses treatment effectswere expressed as log OR with 95% confidenceintervals, and interpreted according to theirstatistical significance assessed at the 5% levelusing a two-sided test.Distributions of results were consideredgraphically using dotplots, and through statisticssummarising location [the ‘average OR’, computed69© Queen’s Printer and Controller of HMSO 2003. All rights reserved.