What Works for Children with Literacy Difficulties? - Digital ...

How should this mass of comparative detail on impact measures be interpreted?
The first thing to be said is that, given the uneven quality of the description, analysis and
reporting of the studies, interpretation needs to be cautious and tentative. It is not the case
that some schemes have been proven effective, and others ineffective, without qualification.
High RGs and effect sizes do show that the relevant approaches have worked for some
children in some circumstances, and may work for others, if implemented with similar care in
similar circumstances. Low RGs and effect sizes show only that the relevant approaches have
not worked for some children in some circumstances, and have no implications for the future,
but they might work for other children in different circumstances.
That said, from inspection of the data and from the wider literature, it has been deduced that
- RGs of exactly 1.0 represent standard progress, or ‘holding one’s own’. Anything above
this represents better than standard progress (but see the next point), while anything less
means that the children are slipping (further) behind;
- RGs below 1.4, and effect sizes below 0.25, represent an impact that does not seem
educationally significant. Pupils in these schemes did not just stay where they were, and
did make some progress, in absolute terms; but it was slow, and they made little or no
relative progress compared to control groups receiving no special intervention. Thus
schemes (or conditions within schemes) with impact measures of this order did not seem
to produce any impact over and above ordinary teaching, unless it is argued that ‘holding
their own’ was a good result for such children - in other words, that without the
intervention they would have fallen even further behind. Schemes in this group may be
considered to have been ‘less effective’;
- all RGs above 1.4, and almost all effect sizes above 0.25, represent impact that is at least
satisfactory, and in some cases excellent. Schemes in this group may be considered to
have been ‘more effective’.
Given this broad distinction, there are a few discrepancies between the RG and effect size lists
for reading (not for spelling). Within Reading Intervention (original), the reading-only
condition (AI1) had low RGs for accuracy and comprehension, and the phonology-only
condition (AI2) had a low RG for accuracy, while the effect sizes seemed satisfactory or even
high. Since the statistical analyses in the original report showed that neither AI produced
greater gains than the control condition, that finding and the RGs are taken here to be the
more accurate. Similarly, in BRP in Worcestershire the phase 1 effect size was low but the
RG was satisfactory, and again the RG is taken to be more accurate.
The RG list for reading contains few values below 1.0 (‘normal progress’), and all but a few
of those RGs arose from control groups. This finding is, however, circular: children receiving
ordinary teaching mostly made the progress to be expected of children receiving ordinary
teaching. What is more interesting is that one control group (in Paired Reading) had RGs
above 1.4, in fact above 2.0, and these children were therefore making better than expected
progress despite, apparently, receiving no extra intervention. Perhaps Paired Reading affected
a high proportion of the schools in the area in which it took place, and therefore the
experimental schools were observed by others, and influenced non-participating schools to
raise their game too. If this is true, it would be an argument for implementing initiatives at a
fairly high density (though it would play even more havoc with evaluation statistics).
145