Final Report of the Evaluation of the Pupil Learning Credits Pilot ...

29
3.3 Statistical analyses of national pupil-level data
Two sets of quantitative analyses were carried out and these are reported in the two
sections that follow. 31 The two methods used differ in a variety of ways and also
differ in some of their findings. Some of the main characteristics of the two
approaches are therefore summarised below.
The first set of analyses use multi-level modelling. This technique is a form of
multiple regression designed to take account of the fact that pupils are clustered
within schools and within LEAs. In this instance the analysis examines whether,
controlling for other factors (including, for example, pupils’ prior attainment), pupils
attending schools receiving PLC funds achieved significantly different test scores
from pupils in other schools.
Test results for 2001, 2002 and 2003 are examined for the analysis of progress to KS3
and results for 2002 and 2003 only for progress to GCSE. For both sets of analyses
the models include a rich set of control variables derived from PLASC data for those
years. Owing to the rich set of control variables the final model reported for most of
the outcome variables accounts for more than 85 per cent of the variation between
schools. All schools in England are included in this analysis.
The second set of analyses use a ‘difference-in-differences’ approach. The technique
is again a form of multiple regression but addresses whether the difference between
attainment in schools receiving PLC funds and attainment in other schools is greater
in 2003 than it was in 2001. The analysis reported focuses only on attainment at KS3.
The analysis uses a slightly less rich array of control variables than the multi-level
modelling. However, in addition, the analysis includes a separate variable for each
individual school included in the analysis which, in effect, takes account of any
differences between schools that may affect attainment and that persisted from 2001
to 2003 but that are not taken into account by the control variables (school ethos
might be such an example).
For this analysis the comparison group is schools in Phase 1 or Phase 2 Excellence in
Cities areas but not in receipt of PLC funding. Thus, whereas the multi-level
modelling analysis compares PLC schools with all schools in England, the differencein-differences
analysis compares performance in PLC schools with that in other urban
schools.
31
See also Kendall et al. (2005) for statistical analyses relating to the evaluation of Excellence in
Cities.