Evaluation of the Two Year Key Stage 3 Project - Communities and ...

ANNEX C - OPTIONAL TEST MODELLINGMulti-level modelling was used as the method of analysis for the Optional Test results. This is a formof multiple regression. Multiple regression enables us to examine the relationship between thedependent variable of interest (for example the Year 7 Mathematics Optional Test score) and a set ofindependent (predictor) variables. These independent variables include whether the pupils attends aProject or Comparison school and also various background characteristics that have previously beenshown to be associated with attainment (for example prior attainment, the sex of the pupil andeligibility for free school meals). ‘Multi-level’ modelling takes into account the fact that pupils areclustered within schools. Failure to take this clustering into account may result in statisticallysignificant associations being reported between the dependent variable (e.g., mathematics test scores)and an independent variable (e.g., being eligible for free school meals) when such a conclusion is notmerited.The models also allow for the possibility that, for example, at some schools pupils with higher levels ofprior attainment may make greater progress than their schoolmates while at other schools it may bepupils with lower prior attainment who make relatively greater progress.It should be remembered that the range of possible values differs for the independent variables listed inTables C1 to C11. For example, the Key Stage 2 Mathematics variable reported in Table C has astandard deviation of more than 20 points whereas the school’s percentage eligible for free schoolmeals has a standard deviation of only 11 and the eligible for free school meals variable may either takethe value of zero (for those not eligible) or one (for those eligible).One means of comparing the ‘effect sizes’ for different independent variables is to express thecoefficients for continuous variables (such as Key Stage 2 score) in terms of the standard deviation ofthat independent variable and to express changes in the dependent variable in terms of its standarddeviation. In the case of continuous variables this enables us to compare the effect size for each variablefor a comparable change in the value of those variables. That is, even though Key Stage 2 Mathematicsscore is measured in marks (from 1-100) and the number of books in the home on a different scale(from 1-6), we are able to compare the two by expressing both in terms of their own standard deviation(or, as is shown in the figures, for a change of 1.41standard deviations). In the case of the dichotomousvariables such as ‘Project school’ the figures indicate the change in Optional Test scores (expressed interms of their standard deviation) associated with having that characteristic (that is, for example, beinga girl or attending a Project school).Such effect sizes are represented in Figures C1 to C4 as are the degree of precision for each of theestimated values with the lines showing the 95% Confidence Intervals for each coefficient estimated.55