Final Report of the Evaluation of the Pupil Learning Credits Pilot ...

More documents

Recommendations

Info

29 3.3 Statistical analyses of national pupil-level data Two sets of quantitative analyses were carried out and these are reported in the two sections that follow. 31 The two methods used differ in a variety of ways and also differ in some of their findings. Some of the main characteristics of the two approaches are therefore summarised below. The first set of analyses use multi-level modelling. This technique is a form of multiple regression designed to take account of the fact that pupils are clustered within schools and within LEAs. In this instance the analysis examines whether, controlling for other factors (including, for example, pupils’ prior attainment), pupils attending schools receiving PLC funds achieved significantly different test scores from pupils in other schools. Test results for 2001, 2002 and 2003 are examined for the analysis of progress to KS3 and results for 2002 and 2003 only for progress to GCSE. For both sets of analyses the models include a rich set of control variables derived from PLASC data for those years. Owing to the rich set of control variables the final model reported for most of the outcome variables accounts for more than 85 per cent of the variation between schools. All schools in England are included in this analysis. The second set of analyses use a ‘difference-in-differences’ approach. The technique is again a form of multiple regression but addresses whether the difference between attainment in schools receiving PLC funds and attainment in other schools is greater in 2003 than it was in 2001. The analysis reported focuses only on attainment at KS3. The analysis uses a slightly less rich array of control variables than the multi-level modelling. However, in addition, the analysis includes a separate variable for each individual school included in the analysis which, in effect, takes account of any differences between schools that may affect attainment and that persisted from 2001 to 2003 but that are not taken into account by the control variables (school ethos might be such an example). For this analysis the comparison group is schools in Phase 1 or Phase 2 Excellence in Cities areas but not in receipt of PLC funding. Thus, whereas the multi-level modelling analysis compares PLC schools with all schools in England, the differencein-differences analysis compares performance in PLC schools with that in other urban schools. 31 See also Kendall et al. (2005) for statistical analyses relating to the evaluation of Excellence in Cities.
30 Multi-level modelling Summary The multi-level modelling analysis examined the attainment of pupils nationally, controlling for a wide range of characteristics including prior attainment. Some statistically significant and negative relationships were identified between attending a PLC school and Key Stage 3 test results. Key Stage 2 test scores improved more rapidly from 2001 to 2003 than did Key Stage 3 test scores. Thus against this backdrop of relative lack of improvement, the decline in pupils’ progress from 2001 to 2003 was more marked in PLC schools than in non-PLC schools (0.7 of a month less progress than expected in Mathematics in 2002, and in 2003 0.5 of a month less progress than expected in KS3 average score and 1.1 of a month less in Science). The value-added analysis of GCSE examinations found some positive associations between attending a PLC school and some of the outcome measures examined. Focusing on progress made from the end of KS3 to GCSE (which coincides with the timing of the PLC pilot scheme), the odds of pupils achieving five or more GCSE passes at grades A* to C in 2003 were significantly improved for those attending schools participating in the pilot scheme (by 21 per cent when considering only the 2003 GCSE results dataset and by 17 per cent when examining the combined 2002 and 2003 dataset). Introduction In this section, key findings from an analysis of the national pupil-level (or valueadded) dataset are presented using multi-level modelling (MLM). Pupils’ academic progress at the end of Year 9 and Year 11 are examined. Pupil-level analysis from KS2 to KS3 The following analysis is based on the Key Stage 2 to Key Stage 3 national valueadded results for pupils who took Key Stage 3 national tests in 2001, 2002 and 2003. The analysis involved two datasets: • 2003 KS3 outcomes only; and • 2001, 2002 and 2003 KS3 outcomes. For both datasets, full prior attainment and PLASC data were available. The inclusion of PLASC data allowed the model to control for a range of pupil background factors (such as ethnicity, known eligibility for Free School Meals and special educational needs status). In addition, the second dataset allowed for the investigation of changes between 2001, 2002 and 2003. Analysis was carried out using multilevel modelling with the following outcome measures as dependent variables, measured in TGAT 32 months of progress: 32 The TGAT (Task Group on Assessment and Testing) months of progress measure is based on the notion that an average pupil makes half a level progress each year and is computed using the following relationship: 1 level = 6 points = 24 months.
Page 1 and 2: RESEARCH Final Report of the Evalua
Page 3 and 4: 2 TABLE OF CONTENTS EXECUTIVE SUMMA
Page 5 and 6: 4 where pupils in PLC schools repor
Page 7 and 8: 6 1. INTRODUCTION The Pupil Learnin
Page 9 and 10: 8 2. METHODS This report draws toge
Page 11 and 12: 10 As part of the evaluation of Exc
Page 13 and 14: 12 curricular activities, departmen
Page 15 and 16: 14 in ten respondents stated that t
Page 17 and 18: 16 Main strengths and weaknesses of
Page 19 and 20: 18 not only intended to assess whet
Page 21 and 22: 20 • pupils in PLC schools report
Page 23 and 24: 22 School experience and behaviour
Page 25 and 26: 24 Around four out of ten pupils re
Page 27 and 28: 26 We examined whether there were s
Page 29: 28 Aspirations for the future Pupil
Page 33 and 34: 32 Pupils’ progress from KS2 to G
Page 35 and 36: 34 Difference-in-differences and ec
Page 37 and 38: 36 when using the discrete measures
Page 39 and 40: 38 4. SUMMARY AND POLICY IMPLICATIO
Page 41 and 42: 40 In particular, the PLC pilot sch
Page 43 and 44: 42 MCNALLY, S. (2005) Economic Eval
Page 45 and 46: ANNEX B CHARACTERISTICS OF PLC SURV

Final Report of the Evaluation of the Pupil Learning Credits Pilot ...

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?