Analytic Culture in the U.S. Intelligence Community (PDF) - CIA

CHAPTER TWELVE
Effect size is usually defined as the difference between the means of two
groups divided by the standard deviation of the control group ⎛ Χ−Χc ⎞ .2
⎜ ∆= e ⎟
⎝ σ c
⎠
Effect sizes calculated in this way estimate the difference between two
group means measured in control group standard deviations as seen in the figure
above. Glass et al. suggest that the choice of the denominator is critical
and that choices other than the control group standard deviation are defensible.
3 However, they endorse the standard choice of using the control group
standard deviation.
Alternatively, Hedges and Olkin show that, for every effect size, both the
bias and variance of its estimate are smaller when standard deviation is
obtained by pooling the sample variance of two groups instead of using the
control group standard deviation by itself. 4 An effect size based on a pooled
standard deviation estimates the difference between two group means measured
in standard deviations estimated for the full population from which both
⎛ Χe − Χc ⎞
experimental and control groups are drawn: ⎜ g = ⎟ , 5 where
⎝ S ⎠
2 2 6
S is the pooled standard deviation: (Ne − 1)(Se ) + (Nc − 1)(Sc )
S = .
e + N Nc − 2
Most commentators suggest that effect sizes can be treated as descriptive
statistics and entered into standard tests for statistical significance. Hedges
and Olkin have shown that the error variance around estimates of effect size is
inversely proportional to the sample size of the studies from which the effect
sizes are drawn. If the effect size in any review is drawn from studies employing
widely different sample sizes, then the heterogeneity of variance among
effect sizes prohibits their use in conventional t-tests, analyses of variance,
and other inferential tests. This is the case in most of these reviews; therefore,
effect sizes reported in this study are treated only with descriptive statistics.
The effect sizes for computer-based training range from 0.20 to 0.46 depending
on the population. 7 The effect size for distance instruction (television) is
Control Mean, σc = Control
Standard Deviation
3
Gene Glass, Barry McGaw and Mary Lee Smith, Meta-Analysis in Social Research.
4
Larry Hedges and Ingram Olkin, Statistical Methods for Meta-Analysis.
5
g=Hedge's Effect Size, S=Hedge's Pooled Standard Deviation
6
Ne=Number of experimental subjects, Nc=Number of control subjects, Se=Standard deviation
of experimental group, Sc=Standard deviation of control group
2
∆=Glass's Effect Size, Χe = Experimental Mean, Χc =
130