Deliverable D 1.3 RESULTS OF META-ANALYSIS ... - cast-eu.org

More documents

Recommendations

Info

Campaigns and awareness raising strategies in traffic safety — <strong>Deliverable</strong> D-1.16. Use differences between overall effects for different sub-groups,together with known theory and results from past analyses, to develophypotheses about the causes of systematic variation in campaigneffect.7. Test hypotheses by using them to inform a model for meta-regressionanalysis, and thereby accept or reject ideas about the partial effects ofdifferent campaign elements.These new steps were followed and refined during the recent INFOEFFEKTproject, which estimated the effect of campaigns on accident counts (Vaa etal, 2004). The criteria employed for study selection under INFOEFFEKT werenarrow, such that the number of campaign effects selected for the final setsummarized was low. CAST represents the first use of the refined method ona substantially broader dataset.In order to fully understand the meaning of the results of meta-analysis, it isimportant to appreciate the process used to generate them. A more detaileddescription of the above steps is therefore given below. The process is alsoexemplified in detail using an effect taken from the CAST database in section7.3.Step 1. The log-odds methodOdds ratios are convenient ways of expressing effects in terms of before-andafteror before-and-during measures (see 7.3 for a fuller explanation). Thereare several reasons to transform odd ratios to their natural logarithms, but insimple terms such transformation helps address a statistical demand, that theset of effects should be normally distributed.Step 2. Publication biasPublication bias describes a tendency for authors and editors to publish onlythose studies demonstrating desirable, statistically significant effects. To put itanother way, it is a tendency to avoid publishing those studies that fail todemonstrate desirable effect. There is documented evidence for publicationbias (Light & Pillemer, 1984), and it means that any set of effects gatheredfrom accessing a selection of available studies will be incomplete in that it willlack those undesirable or non-significant effects that were never published.The problem is that the true overall effect cannot be derived from sample ofeffects that does not fairly represent the whole set of effects. Fortunately,statistical tests are now available that can be used to detect publication biasfor a sample of effects. Such tests are used on the set of effects collected forCAST.Several methods have been developed to correct the overall effect of a set ofeffects for which publication bias has been indicated. The ’trim-and-fill’ methodproposed by Duval and Tweedie (2000a & b) has been developed to test and,where such bias is indicated, adjust the overall effect estimate by generatingthe missing, ‘undesired’ effects to complete the original set of effects. A new,corrected overall effect estimate is then calculated, based on the newcompleted set of effects. This is exemplified in section 7.3.Step 3: Fixed- and random-effects: homogeneity vs heterogeneity.50
Campaigns and awareness raising strategies in traffic safety — <strong>Deliverable</strong> D-1.1One must decide whether to use a so-called fixed-effects or random-effectsmodel when carrying out meta-analysis. A fixed-effects model assumes thatthe same intervention (in our case a road safety campaign) will have the sameeffect regardless of whether it is carried out on different populations, atdifferent times or in different countries. That is, it is assumed that thecharacteristics of the intervention are such that it will have the same effect indifferent contexts. Any variation that is observed will just be due to chance. Arandom-effects model, on the other hand, assumes that characteristics of thesituation, as well as the intervention itself, can cause variations in effect, asindeed can characteristics of the method used to measure the effect.Fortunately again, a statistical test can be used to inform the decision aboutwhich model will give the best estimate of overall effect for a set of effects.The test assesses whether there is significant heterogeneity among the set ofeffects (Everitt, 2002). Non-significant heterogeneity implies homogeneity, inwhich case a fixed-effects model is therefore used to describe the overalleffect. Where the level of heterogeneity is significant, a random-effects modelis used, and an estimator is introduced to account for possible unknownsources of systematic variation between individual effects.The true overall effect of a heterogeneous set of effects can be inaccuratelydescribed using a fixed-effects model, whereas the true overall effect of ahomogeneous set of effects will be estimated quite accurately by both fixedandrandom-effects models. However, because the random-effects modelreports larger confidence intervals, reflecting the greater uncertainty assumed,it is preferable to use a fixed-effects model on a homogenous dataset.Step 4. Conservative best estimate of overall effect.Using the preceding steps it is possible to generate estimates of overall effectin four different ways:(1) using a fixed-effects model without adjusting for publication bias;(2) using a fixed-effects model and adjusting for publication bias;(3) using a random-effects model without adjusting for publication bias;or(4) using a random-effects model and adjusting for publication bias.The statistical output on the level of publication bias and heterogeneity in thedataset is assessed together in order to select the best estimate for anintervention’s overall effect. In this way the statistical tests help answerquestions of generalizability. They address to what extent, based on the set ofeffects identified, one can expect to achieve an overall effect in differentsituations, and therefore what level of confidence should be employed todescribe the overall effect. They essentially allow us to make more usefulpredictions about future situations.The following thus summarizes the strategy followed in the analysis andpresentation of results:(1) always use the random-effects model where heterogeneity isindicated;51
Page 1 and 2: Campaigns and awareness raising str
Page 49: Campaigns and awareness raising str
Page 101 and 102:
Campaigns and awareness raising str
Page 103 and 104:
Page 105 and 106:
Page 107 and 108:
Page 109 and 110:
Page 111 and 112:
Page 113 and 114:
Page 115 and 116:
Page 117 and 118:
Page 119 and 120:
Page 121 and 122:
Page 123 and 124:
Page 125 and 126:
Page 127 and 128:
Page 129 and 130:
Page 131 and 132:
Page 133 and 134:
Page 135 and 136:
Page 137 and 138:
Page 139 and 140:
Page 141 and 142:
Page 143 and 144:
Page 145 and 146:
Page 147 and 148:
Page 149 and 150:
Page 151:
show all

Deliverable D 1.3 RESULTS OF META-ANALYSIS ... - cast-eu.org

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?