The Interactive Whiteboards, Pedagogy and Pupil Performance ...

8.1.4 Difference-in-Differences (Between-Departments Effects)
The final approach to evaluating the effect of the installation of IWBs is to test
whether the department in the school that saw the largest increase in IWB
installations between 2004 and 2005 also achieved the largest gains in pupil
performance (compared to the other core subjects). This requires us to compare
changes in the effectiveness of the Maths, English and Science departments in a
school, and so we need to adjust the pupil test scores to reflect the fact that the
exam sat in each subject is measured on a different scale.
The first stage is to calculate an adjusted achievement score for each child, which
reflects the child’s achievement in a subject relative to the mean child with the same
prior Key Stage mark and socio-demographic characteristics:
adjusted yist
= yist
− X ˆ
istδ = M
st
β + Z
st
χ + γ
s
+ ε
(7)
ist
1 4 44 2 4 4 43
school fixed effect dummy
Twelve separate regressions are run to calculate the adjusted pupil achievement = 3
subjects x 2 years of data x 2 Key Stages. The adjusted pupil achievement is then
standardised for each of the 12 regressions as a z-score with mean of zero and a
standard deviation of one.
The school mean adjusted achievement, y dst , for each subject, d, in each cohort is
calculated as the average of the pupil adjusted achievement z-scores. y dst reflects
the departmental effort and any benefit from time-invariant resources in the
department (γ s ), any benefit from a cohort peer group effect (Z st ) and any benefit
from IWBs (M st ).
We compare the change in mean adjusted achievement in the three departments in
the school with the change in IWB installations that the department experienced
using the following first difference model:
yds, t
− yds,
t−1
= ( M
ds,
t
− M
ds,
t−1
) β + eng + sci + ( Z
s,
t
− Z
s,
t−1
)^ eng + ( Z
s,
t
− Z
s,
t−1
)^ sci (8)
+ ( M
ds,
t
− M
ds,
t−1
)^ eng + ( M
ds,
t
− M
ds,
t−1
)^ sci + γ + ε
Equation (8) allows us to identify the effect of the treatment, controlling for changes
in the school characteristics between 2004 and 2005 using school fixed effects. It
allows for time-trends to vary by subject using English and Science dummies, though
we expect them to be zero since z-scores were created for each subject. It also
allows any effect of a change in peer group in the school between the two years to
differ by subject and allows for the possibility that the effect of the change in IWB
installation on pupil outcomes differs systematically between subjects.
A much simpler model is also tested, where the effect of a change in IWB installation
is assumed to equal across subjects, as is the change in peer group:
yds
t
− yds
t− = ( M
ds t
− M
ds t−
) β + eng + sci + γ
s
+ ε
(9)
, , 1
, , 1
dst
Results – Between-Departments Effects
Table 11 shows the regressions for Key Stage 3 for the specifications shown in
equation (9) and equation (8). The effect of the change in IWB installations has no
significant relationship with the relative progress in effectiveness of a department,
compared to other departments in the schools who did not experience the large
increase in IWB installations. In the more complex regression it is possible to see
s
dst
70