RESEARCH METHOD COHEN ok

EVIDENCE-BASED EDUCATIONAL RESEARCH AND META-ANALYSIS 293
2 Coding the study characteristics (e.g. date,
publication status, design characteristics,
quality of design, status of researcher).
3 Measuring the effect sizes (e.g. locating the
experimental group as a z-score in the control
group distribution) so that outcomes can be
measured on a common scale, controlling for
‘lumpy data’ (non-independent data from a
large data set).
4 Correlatingeffectsizeswithcontextvariables
(e.g. to identify differences between wellcontrolled
and poorly-controlled studies).
Effect size (e.g. Cohen’s d and eta squared) are
the preferred statistics over statistical significance
in meta-analyses, and we discuss this in Part Five.
Effect size is a measure of the degree to which a
phenomenon is present or the degree to which
anullhypothesisisnotsupported.Wood(1995:
393) suggests that effect-size can be calculated
by dividing the significance level by the sample
size. Glass et al.(1981:29,102)calculatetheeffect
size as:
(Mean of experimental group − mean of control group)
Standard deviation of the control group
Hedges (1981) and Hunter et al. (1982) suggest
alternative equations to take account of
differential weightings due to sample size
variations. The two most frequently used indices
of effect sizes are standardized mean differences
and correlations (Hunter et al.1982:373),though
non-parametric statistics, e.g. the median, can
be used. Lipsey (1992: 93–100) sets out a series
of statistical tests for working on effect sizes,
effect size means and homogeneity. It is clear
from this that Glass and others assume that metaanalysis
can be undertaken only for a particular
kind of research – the experimental type – rather
than for all types of research; this might limit its
applicability.
Glass et al. (1981)suggestthatmeta-analysis
is particularly useful when it uses unpublished
dissertations, as these often contain weaker correlations
than those reported in published research,
and hence act as a brake on misleading, more
spectacular generalizations. Meta-analysis, it is
claimed (Cooper and Rosenthal 1980), is a means
of avoiding Type II errors (failing to find effects
that really exist), synthesizing research findings
more rigorously and systematically, and generating
hypotheses for future research. However, Hedges
and Olkin (1980) and Cook et al. (1992: 297)
show that Type II errors become more likely as the
number of studies included in the sample increases.
Further, Rosenthal (1991) has indicated a
method for avoiding Type I errors (finding an
effect that, in fact, does not exist) that is based on
establishing how many unpublished studies that
average a null result would need to be undertaken
to offset the group of published statistically
significant studies. For one example he shows a
ratio of 277:1 of unpublished to published research,
thereby indicating the limited bias in published
research.
Meta-analysis is not without its critics (e.g. Wolf
1986; Elliott 2001; Thomas and Pring 2004). Wolf
(1986: 14–17) suggests six main areas:
It is difficult to draw logical conclusions
from studies that use different interventions,
measurements, definitions of variables, and
participants.
Results from poorly designed studies take their
place alongside results from higher quality
studies.
Published research is favoured over unpublished
research.
Multiple results from a single study are used,
making the overall meta-analysis appear more
reliable than it is, since the results are not
independent.
Interaction effects are overlooked in favour of
main effects.
Meta-analysis may have ‘mischievous consequences’
(Wolf 1986: 16) because its apparent
objectivity and precision may disguise procedural
invalidity in the studies.
Wolf (1986) provides a robust response to
these criticisms, both theoretically and empirically.
Wolf (1986: 55–6) also suggests a ten-step
sequence for carrying out meta-analyses rigorously:
1 Make clear the criteria for inclusion and
exclusion of studies.
Chapter 13