What Works for Children with Literacy Difficulties? - Digital ...

Secondly, most of the tests provided only reading/spelling age data and not standardised
scores. Though apparently easier to interpret, reading and spelling ages are statistically
unsatisfactory - for example, establishing whether a gain in test scores is statistically
significant is more difficult for reading and spelling ages than for standardised scores.
Reading and spelling age data do allow the calculation of the ratio gain - but this is in itself
not a very useful statistic, especially for low-attaining groups. Pupils in such groups might
not be expected to make a month’s gain in reading or spelling age in one calendar month, for
perfectly valid reasons. Standardised scores allow much more direct comparisons of amount
of gain. Ratio gains have nevertheless been used in much of this analysis because for most of
the studies they were the only impact measure which could be calculated.
Thirdly, for many of the tests used it was impossible to calculate effect sizes, which are
statistically much more satisfactory than ratio gains. If a standardised test is used, an effect
size can be calculated even in the absence of an explicit control group; but if a nonstandardised
test is used then an effect size can be calculated only if control group data,
including the standard deviation, are reported.
More generally, it is noteworthy that the evaluations included here number just 25. As pointed
out in chapter 1, they represent more studies and approaches than that, because for Paired
Reading and Parental Involvement in particular one study stands for many, and because
several studies contained control groups and/or alternative interventions. But the total is still
not impressive, and it is to be hoped that anyone currently devising an intervention will
automatically consider the necessity for, and commission, an evaluation. The Government is
setting an example here, with its evaluations of Summer Literacy Schools and of the National
Literacy Strategy.
4.3 Recommendations
Whenever an educational innovation is devised and tried out, an evaluation should be
commissioned.
All evaluations should be based on the gathering of quantitative attainment data, and the data
should come from the use of standardised tests, and not non-standardised instruments such as
reading- and spelling-age tests.
Properly defined control groups should be set up, through random assignment or at least by
matching.
All evaluations should report as a minimum the following information:
• the date when the evaluation was carried out (in addition to the date of reporting)
• the exact age-range of the children involved
• salient characteristics of the children involved, for example whether they had special
educational needs
• the numbers of children in the experimental and control groups and in any alternative
intervention groups
61