What Works for Children with Literacy Difficulties? - Digital ...

Where there is a control group but it is not properly equivalent to the experimental group,
its data cannot be reliably used in these calculations. In this analysis, where a control
group was not properly constituted, generally no attempt has been made to calculate an
effect size from its data. An exception has been made for the Paired Writing studies.
Even where the control group was properly set up the choice of s.d. to use as the divisor in
the formula is still problematic, since authorities in the field seem to advocate or adopt
different approaches:
(a) Glass, McGaw and Smith (1981) seem to recommend the s.d. of the control group’s
post-test scores (e.g. p.114), and counsel against pooling it with any other s.d. (p.106).
(b) Topping and Lindsay (1992, p.211) give a formula using the s.d. of the control
group’s gain. Very few reports state this statistic, but it can be calculated from the s.d’s of
the control group’s pre- and post-test scores, where both are given. However, effect sizes
based on this statistic measure something rather different from those based on the other
forms of s.d. mentioned here, and (depending on the size of the correlation between pre-
and post-test scores) such effect sizes may be overestimated by a factor between about 1.3
and 2.2.
(c) NCET (1996, pp.6, 10) in the ILS evaluation used a slightly different formula, with
the divisor being (apparently) the arithmetic mean of the pre-test s.d’s of experimental and
control groups.
(d) Where the control group’s post-test s.d. is significantly larger than that group’s pretest
s.d., the post-test s.d. may be inappropriate as a measure of the variance of the
population of interest. In these circumstances, the control group’s protest s.d. may be the
appropriate divisor.
In practice, for this report, much of the debate over which s.d. to use became irrelevant,
since only a few reports stated more than one form of s.d. (and some stated none at all).
The following policy on effect sizes was therefore adopted:
• where an evaluation report did not give the data needed to calculate an effect size, we
state that ‘Effect sizes were not given and could not be calculated’;
• where an evaluation report stated an effect size it was accepted, however calculated;
• where no effect size was given but the information to calculate it, according to any of
the above methods using a reliable control group s.d., was given, it was calculated.
There were a few studies in which, though effect sizes were not stated and could not be
calculated, they would clearly have been close to zero and non-significant, because the
gain made by an experimental or alternative intervention group was not significantly
greater than that made by the control group. Where this was the case, it has been stated
and the information has been taken as reliable.
In a few cases, an effect size has been reported where there was no control group and the
authors of the study did not report one. These are all studies which used standardised
75