RESEARCH METHOD COHEN ok

292 EXPERIMENTS AND META-ANALYSIS
effect size, that is to say, in terms of how much
difference they make rather than only in terms
of whether or not the effects are statistically
significant at some arbitrary level such as 5 per
cent. Because, with effect sizes, it becomes easier to
concentrate on the educational significance of a
finding rather than trying to assess its importance
by its statistical significance, we may finally see
statistical significance kept in its place as just one
of many possible threats to internal validity. The
move towards elevating effect size over significance
levels is very important (see also Chapter 24), and
signals an emphasis on ‘fitness for purpose’ (the
size of the effect having to be suitable for the
researcher’s purposes) over arbitrary cut-off points
in significance levels as determinants of utility.
The term ‘meta-analysis’ originated in
1976 (Glass 1976) and early forms of meta-analysis
used calculations of combined probabilities and
frequencies with which results fell into defined
categories (e.g. statistically significant at given
levels), although problems of different sample sizes
confounded rigour (e.g. large samples would yield
significance in trivial effects, while important data
from small samples would not be discovered because
they failed to reach statistical significance)
(Light and Smith 1971; Glass et al. 1981;McGaw
1997: 371). Glass (1976) and Glass et al. (1981)
suggested three levels of analysis:
primary analysis of the data
secondary analysis, a re-analysis using different
statistics
meta-analysis analysing results of several
studies statistically in order to integrate the
findings.
Glass et al.(1981)andHunteret al. (1982) suggest
eight steps in the procedure:
1 Identify the variables for focus (independent
and dependent).
2 Identify all the studies which feature the
variables in which the researcher is interested.
3 Code each study for those characteristics
that might be predictors of outcomes and
effect sizes. (e.g. age of participants, gender,
ethnicity, duration of the intervention).
4 Estimate the effect sizes through calculation
for each pair of variables (dependent
and independent variable) (see Glass 1977),
weighting the effect-size by the sample size.
5 Calculate the mean and the standard
deviation of effect-sizes across the studies, i.e.
the variance across the studies.
6 Determine the effects of sampling errors,
measurement errors and range of restriction.
7 If a large proportion of the variance is
attributable to the issues in Step 6, then
the average effect-size can be considered an
accurate estimate of relationships between
variables.
8 If a large proportion of the variance is not
attributable to the issues in Step 6, then
review those characteristics of interest which
correlate with the study effects.
Cook et al. (1992:7–12)setoutafivestep
model for an integrative review as a research
process, covering:
1 Problem formulation, where a high quality
meta-analysis must be rigorous in its attention
to the design, conduct and analysis of the
review.
2 Datacollection,wheresamplingofstudiesfor
review has to demonstrate fitness for purpose.
3 Data retrieval and analysis, where threats
to validity in non-experimental research – of
which integrative review is an example – are
addressed. Validity here must demonstrate
fitness for purpose, reliability in coding, and
attention to the methodological rigour of the
original pieces of research.
4 Analysis and interpretation, where the
accumulated findings of several pieces of
research should be regarded as complex
data points that have to be interpreted by
meticulous statistical analysis.
Fitz-Gibbon (1984: 141–2) sets out four steps
in conducting a meta-analysis:
1 Finding studies (e.g. published, unpublished,
reviews) from which effect sizes can be
computed.