Final Report of the Evaluation of the Pupil Learning Credits Pilot ...

34
Difference-in-differences and economic evaluation
Summary
The economic evaluation of the PLC pilot scheme involved an analysis of the scheme’s
impact on attainment in Mathematics and English at the end of KS3 and on a measure of
attendance at school. A difference-in-differences analysis compared outcomes for pupils in
PLC schools with those in a comparison group of schools before (2001) and after the pilot
scheme (2003).
Two groups of comparison schools were used: non-PLC schools in EiC Phase 1 areas; and
non-PLC schools in EiC Phase 1 and Phase 2 areas. Results were very similar for the two
comparison groups.
The most detailed specification (with controls for prior attainment, some pupil characteristics
and a broad range of school characteristics) shows that the proportion of pupils achieving
level 5 or higher in Mathematics was 2.1 percentage points higher in PLC schools. There
was also a (slightly smaller) impact on attaining level 4 or above. No statistically significant
results were found in relation to attainment in English.
The effect of the PLC pilot scheme on the average level attained in Mathematics was also
examined. It was found that involvement in the scheme increased the number of pupils
moving up a level in PLC schools by 6.4 percentage points. There was no effect on the
average level attained in English.
When examining school-level absences, some evidence of an effect of PLCs is found, with
the pilot scheme reducing the percentage of half days missed by about half a percentage
point.
A cost-benefit analysis which seeks to predict the impact of the effects described above on
labour market outcomes in later life was carried out. The analysis necessitates a series of
strong assumptions and thus the results have to be viewed with some care. It is suggested
that the pilot scheme is potentially cost-effective, which in turn appears to be driven by the
low cost of the scheme.
Introduction
The economic evaluation of Pupil Learning Credits pilot scheme involves an analysis
of whether the scheme had any impact on attainment at the end of Key Stage 3 in
English or Mathematics and on a measure of attendance at school (see McNally,
2005). The possible impact of any resulting increase in attainment on subsequent
labour market earnings is also being considered. This cost-benefit analysis is
necessarily crude as we do not know the relationship between attainment at Key Stage
3 (or school attendance) and outcomes in the labour market. Furthermore, even if
such information were available, the children who benefit from PLCs (and related
policies such as Excellence in Cities) are in relatively disadvantaged areas and hence
may be affected differently by an increase in attainment at this time. Ideally, to find
out whether any PLC effect on attainment has an enduring impact, which also affects
labour market outcomes, one would want to follow these pupils over time.
However, the first part of the analysis, which is based on a difference-in-differences
analysis, is built on a firm foundation. This involves comparing the outcomes of