RESEARCH METHOD COHEN ok

294 EXPERIMENTS AND META-ANALYSIS
2 Search for unpublished studies.
3 Develop coding categories that cover the
widest range of studies identified.
4 Lookforinteractioneffectsandexaminemultiple
independent and dependent variables
separately.
5 Testforheterogeneityofresultsandtheeffects
of outliers, graphing distributions of results.
6 Checkfor inter-rater coding reliability.
7 Use indicators of effect size rather than
statistical significance.
8 Calculateunadjusted(raw)andweightedtests
and effects sizes in order to examine the
influence of sample size on the results found.
9 Combinequalitativeandquantitativereviewing
methods.
10 Report the limitations of the meta-analyses
conducted.
One can add to this the need to specify the
research questions being asked, the conceptual
frameworks being used, the review protocols being
followed, the search and retrieval strategies being
used, and the ways in which the syntheses of the
findings from several studies are brought together
(Thomas and Pring 2004: 54–5).
Gorard (2001: 72–3) suggests a four-step model
for conducting meta-analysis:
1 Collect all the appropriate studies for
inclusion.
2 Weight each study ‘according to its size and
quality’.
3 List the outcomemeasures used.
4 Selectamethodforaggregation,basedonthe
nature of the data collected (e.g. counting
those studies in which an effect appeared and
those in which an effect did not appear, or
calculating the average effect size across the
studies).
Evans and Benefield (2001: 533–7) set out six
principles for undertaking systematic reviews of
evidence:
Aclearspecificationoftheresearchquestion
which is being addressed.
Asystematic,comprehensiveandexhaustive
search for relevant studies.
The specification and application of clear criteria
for the inclusion and exclusion of studies,
including data extraction criteria: published;
unpublished; citation details; language; keywords;
funding support; type of study (e.g.
process or outcome-focused, prospective or
retrospective); nature of the intervention; sample
characteristics; planning and processes of
the study; outcome evaluation.
Evaluations of the quality of the methodology
used in each study (e.g. the kind of experiment
and sample; reporting of outcome measures).
The specification of strategies for reducing bias
in selecting and reviewing studies.
Transparency in the methodology adopted for
reviewing the studies.
Gorard (2001) acknowledges that subjectivity can
enter into meta-analysis. Since so much depends
upon the quality of the results that are to be
synthesized, there is the danger that adherents may
simply multiply the inadequacies of the database
and the limits of the sample (e.g. trying to compare
the incomparable). Hunter et al. (1982) suggest
that sampling error and the influence of other
factors has to be addressed, and that it should
account for less than 75 per cent of the variance
in observed effect sizes if the results are to be
acceptable and able to be coded into categories.
The issue is clear here: coding categories have to
declare their level of precision, their reliability
(e.g. inter-coder reliability – the equivalent of
inter-rater reliability, see Chapter 6) and validity
(McGaw 1997: 376–7).
To the charge that selection bias will be as strong
in meta-analysis – which embraces both published
and unpublished research – as in solely published
research, Glass et al. (1981: 226–9) argue that
it is necessary to counter gross claims made in
published research with more cautious claims
found in unpublished research.
Because the quantitative mode of (many)
studies demands only a few common variables