Aggregate versus Disaggregate Data in Measuring School Quality

Outcomes with and without measurement error are compared in order to see if the
aggregate estimator is in fact more robust to errors in measurement than the disaggregate
estimator. The parameters used to randomly generate the number of students in each school are
also changed, to see how variability in school size affects the performance of the estimators.
3. Results
Table 1 shows the first set of results for 1000 samples, each of 100 schools whose size is
distributed lognormal with mean 120 and variance 50000. According to this distribution, about
70% of schools have sizes between 15 and 250 students. As expected, the disaggregate estimator
performs best on almost all measures. The aggregate estimator’s performance, however, is very
good, and clearly above the OLS estimator’s performance. OLS tends to pick small schools as
the top schools. The average school size for the top ten schools as estimated by OLS is about
102, while the true average for this group is 120. OLS estimators are based on residuals whose
2 2
variance is σ + σ / n . So, quality estimates for small schools will have a larger variance and
u
e
will be more likely to be either at the bottom or top of the rankings. However, table 1 shows that
both the aggregate and disaggregate estimators tend to pick large schools as the top schools so
they are also a biased predictor of top schools.
The aggregate and disaggregate estimators have a shrinkage factor reduces the residuals
of small schools. Recall the shrinkage factor is
2
σ u
2 2
σ u + σ e / n
15
. This factor is always less than one, but
decreases with school size, bringing down the absolute value of small school residuals. Results in
table 1 suggest that the shrinkage factor may over-compensate for the residuals effect, and thus,
leave mainly large schools in the extremes. Estimators with a smaller shrinkage factor (the factor