Aggregate versus Disaggregate Data in Measuring School Quality

Aggregate versus Disaggregate Data in Measuring School Quality
Francisca G.-C. Richter Contact Author: B. Wade Brorsen
Department of Economics Department of Agricultural Economics
Cleveland State University Oklahoma State University
Rhodes Tower Room #1704 414 Ag Hall
Cleveland, OH 44115 Stillwater, OK 74078-6026
Phone: (216) 687-4529 Phone: (405) 744-6836
FAX: (216) 687-9206 FAX: (405) 744-8210
e-mail: f.richter16@csuohio.edu e-mail: brorsen@okstate.edu
Francisca G. C. Richer is a lecturer at Cleveland State University. B. Wade Brorsen is regents
professor and Jean & Patsy Neustadt Chair in the Department of Agricultural Economics at
Oklahoma State University.
L:\Home\Brorsen\PAPERS\Oldpapers Feb 05\Measuring School Quality\aggdis1revised.doc 2/9/2006

Aggregate versus Disaggregate Data in Measuring School Quality
Abstract
This article develops a measure of efficiency to use with aggregated data. Unlike the most
commonly used efficiency measures, our estimator adjusts for the heteroskedasticity created by
aggregation. Our estimator is compared to estimators currently used to measure school
efficiency. Theoretical results are supported by a Monte Carlo experiment. Results show that for
samples containing small schools (sample average may be about 100 students per school but
sample includes several schools with about 30 or less students), the proposed aggregate data
estimator performs better than the commonly used OLS and only slightly worse than the
multilevel estimator. Thus, when school officials are unable to gather multilevel or disaggregate
data, the aggregate data estimator proposed here should be used. When disaggregate data are
available, standardizing the value-added estimator should be used when ranking schools.
Keywords: data aggregation, error components, school quality
1

Aggregate versus Disaggregate Data in Measuring School Quality
Over the last three decades, resources devoted to education have continuously increased
while student performance has barely changed (Odden and Clune 1995). In response, several
states now reward public schools that perform better than others, based on their own measures of
school quality (Ladd 1996). Test scores are used not only by policymakers in reward programs
but are also presented in state report cards issued to each school. Already more than 35 states
have comprehensive report cards reporting on a variety of issues including test scores and a
comparison of school variables with district and state averages. But often the information
presented is misleading or difficult to interpret. Accurate information on school performance is
needed if report cards and reform programs are to succeed in improving public schools.
Hierarchical linear modeling (HLM), a type of multilevel modeling, has been recognized
by most researchers as the appropriate technique to use when ranking schools by effectiveness.
As Webster argues, HLM recognizes the nested structure of students within classrooms and
classrooms within schools, producing a different variance at each level for factors measured at
that level. Multilevel data, also called disaggregate data is needed to implement HLM. For
example, two-level data could consist of school-level and student-level variables. The value-
added framework in combination with HLM has become popular among researchers (Hanushek,
Rivkin, and Taylor 1996; Goldstein 1997; Woodhouse and Goldstein 1998). Value-added
regressions isolate a school’s effect on test scores during a given time period, by using previous
test scores as a regressor. As of 1996, among the 46 out of 50 states with accountability systems,
only two used value-added models (Webster et al. 1996). Multilevel analysis has been criticized
for being a complicated statistical analysis that school officials cannot understand (Ladd 1996).
2

Most state school evaluation systems use aggregate data. Rather than having data for each
student within each school, aggregate data provides only averages over all students, within a
school. School administrators may be able to obtain records of each student’s individual test
score but may not be able to match them with their parents’ income, for example. Therefore,
average test scores in a school are matched to the average income in the respective school
district.
To measure school quality with aggregate data, it is common to regress school mean
outcome measures on the means of several demographic and school variables The residuals from
this regression are totally attributed to the school effect, and thus, are used to rank schools.
Although the use of aggregate data has been widely criticized in the literature (Webster et al.
1996; Woodhouse and Goldstein 1998), many states use aggregate data. This article proposes a
new and more efficient estimator of quality based on aggregate data, and compares it with the
commonly used ordinary least squares (OLS) estimator as well as with the value-added-
disaggregate estimator. Estimators based on disaggregate data will perform better than an
estimator based on aggregate data. The questions that arise are: by how much will their
performances differ? Should schools be using OLS, when they can use a more efficient
aggregate estimate at no extra cost?
One of Goldstein’s main oppositions to aggregate data models is that they say nothing
about the effects upon individual students. Also, aggregate data does not allow studying
differential effectiveness, which distinguishes between schools that are effective for low
achieving students and schools that are effective for high achieving students. The inability to
handle differential effectiveness is a clear disadvantage of aggregate as compared to disaggregate
data.
3

Another problem with using aggregate data is that the aggregated variables may not have
been obtained from the same group of students or individuals. Family income, for example, may
be a county average rather than the average over the school. Average test scores and previous
test scores may also come from different groups of students due to student mobility. In some
school districts student mobility can be quite high (Fowler-Finn). Previous test scores are often
not available for students who have changed schools. With disaggregate data, school effects are
often estimated by reducing the sample to those students tested in both periods. Disaggregate
data at least permit a study of mobile students using regressions without previous test scores.
With aggregate data, a percent of students not present in both periods can be included as a
regressor, but that does not fully capture the measurement error in explanatory variables or the
possible differential effectiveness of schools in educating mobile students.
However, when aggregate data are all that schools have, is it still possible to detect the
extreme over and under performing schools? When using OLS on aggregate data, it has been
observed that small schools are disproportionately rewarded (Clotfelter and Ladd 1996). The
estimator proposed here eliminates that bias by using standardized residuals to rank schools.
Woodhouse and Goldstein (1998) argue that residuals from regressions with aggregate
data are highly unstable and therefore, unreliable measures of school efficiency. Woodhouse and
Goldstein analyze an aggregate model used in a previous study and show how small changes in
the independent variables as well as the inclusion of non-linear terms will change the rank
ordering of regression residuals. However, their data set is small and they do not examine
whether disaggregate data would have also led to fragile conclusions.
The past research criticizing aggregate data did not consider maximum likelihood
estimation of the aggregate model. Goldstein (1995), for example, illustrates the instability of
4

aggregate data models with an example in which he compares estimates coming from an
aggregate model versus estimates from several multilevel models and shows they are different.
Goldstein’s (1995) aggregate model, however, does not provide an estimate of the between-
student variance, which suggests that the author does not use MLE residuals to estimate school
effects. Maximum likelihood estimation is possible since the form of heteroskedasticity for the
aggregate model is known (Dickens 1990).
While it is expected that aggregation will attenuate the bias due to measurement error,
few researchers have compared aggregate data models versus multilevel models while
considering measurement error. Hanushek, Rivkin, and Taylor (1996) argue this aggregation
produces an ambiguous bias on the estimated regression parameters. Thus they suggest an
empirical examination of the effects of aggregation in the presence of measurement error.
Although the conventional wisdom is that aggregate data should not be used to measure
school quality, the literature on which this belief is based, is insufficient to support it. Research
comparing aggregate with disaggregate models have used ordinary least squares rather than
maximum likelihood estimators so the validity of their criticism is unclear. Efficient estimators
of school quality based on aggregate data, as well as their confidence intervals will be developed
here and compared to multilevel estimators with and without measurement error. In the process,
a standardized version of the value-added multilevel estimator is proposed. Since many states use
aggregate data to rank and reward schools, the relevance of this issue cannot be denied.
1. Theory
Estimators of school effects on student achievement based on disaggregate data have
been developed and reviewed extensively in the education literature, and are presented only
5

iefly here. However, little effort has been devoted to develop appropriate estimators for
aggregate data.
This section consists of three parts. The first part will show how aggregation of a 2-level
error components model, with heterogeneous number of first-level units within second-level
units, leads to a model with heteroskedastic error terms. Therefore, for estimators of the
parameters of the model to be efficient, ML or GLS estimation is required. The aggregate data
estimator is presented as well as its standardized version.
The second part derives confidence intervals for the aggregate data estimator and presents
the confidence intervals commonly used for disaggregate data. The third part introduces
measurement error in the model and derives the bias of parameter estimates.
1.1. Aggregation of a Simple 2-Level Error Components Model
Consider the following model:
(1) Y ij = ( X β ) ij + u j + eij
, i = 1 , K , n j j = 1,
K J ,
whereY ij is the test score of the i th student in the j th school, ( Xβ ) ij is the fixed part of the model,
likely to be a linear combination of student and school characteristics, such as previous test score
(for a value added measure), parents’ education, and average parents’ income for each school,
u j is the random effect due to school, that we are trying to estimate, and e ij is the unexplained
portion of the test score, with distributions given by
2
2
u ~ iid N(
0,
σ ), e ~ iid N(
0,
σ ), cov( u , e ) = 0 .
j
u
ij
In matrix notation the model is:
(1.a) Y = Xβ
+ Ζ u + e ,
where
e
j
ij
6

2 ⎡σ
e I n
⎢
V = ⎢
⎢
⎣
⎡1
⎤
n 0
1
⎢
⎥
Ζ = ⎢ O ⎥ ,
⎢
⎥
⎣
0 1nJ
⎦
1
( , V )
Ζ u + e ~ N 0 ,
2
+ σ J
0
u
n
1
0 ⎤
⎥
O ⎥ .
2
2
σ I + ⎥
e n σ J
J u nJ
⎦
The random effect u j represents the departure from the overall mean effect of schools on
students’ scores. While the intercept contains the overall mean effect of schools, u j measures by
how much school j deviates from this mean.
The shrinkage estimator of u j is (Goldstein 1995):
2 2 2
j
(2) uˆ
j = ( σ u /( σ u + σ e / n j ))( yˆ
ij ) / n j
y ˆ = Y − ( X ˆ β ) ,
ij
ij
7
n
∑ i=
1
where the yˆ ij ’s are called raw residuals and βˆ is the MLE of β . So the school effect for school j
is estimated by the raw residuals, averaged over all students, and ‘shrunken’ by a factor that is a
function of the variance components and the number of students in the school. The larger the
number of students in a school, the closer this factor is to one. But if school size is small, there
will be less information to estimate the school effect. Thus, the shrinkage factor becomes
smaller, making the estimate of the school effect deviate less from the overall mean.
each school,
ij
Now let us see how the model changes with aggregation. Adding over all students within
n j
n j
n j
∑ Y =
= ∑ ( X
β ) + +
= ij n ju
1
1
j ∑ =
i ij e
i
i 1 ij

and dividing by the number of students in each school, leads to the following model:
(3) j = ( X β ) j + u j + e j , j = 1, K,
J
Y .
.
.
2
2
u iid N(
0,
σ ), e ~ N(
0,
σ / n ), cov( u , e ) = 0 ,
j ~ u . j
e j
j . j
where the dot is the common notation to denote that the variable has been averaged over the
corresponding index; students in this case. The error term for the aggregated model will be
2 2
v ~ ( 0,
σ + σ / n ) .
j
u
e
j
Again, in matrix notation the model is:
(3.a) Y a = X a β + u + e a ,
X
a
⎡ 1 1′
n1
⎢
= ⎢
⎢
⎣
0
n1
V a
0 ⎤ ⎡ 1 1′
⎤ ⎡ 1
n
1′
⎤
n1
1
n1
n1
⎥ ⎢ ⎥ ⎢ ⎥
O ⎥ X , Ya
= ⎢ M ⎥Y
, ea
= ⎢ M ⎥e
1 1′
⎥ ⎢ 1
nJ
nJ
⎦
1′
⎥ ⎢ 1
⎣ nJ
nJ
⎦
1′
⎥
⎣ nJ
nJ
⎦
a
( )
u + e ~ N 0,V ,
2 2 ⎡σ
u + σ e / n1
⎢
= ⎢
⎢
⎣ 0
O
8
a
2
u
0
2
σ + σ / n
We are interested in estimating the random effects u j ’s. For this, we estimate the MLE
residuals of the error term vj. We define our estimator as the conditional mean of u j given vj, i.e.,
u ~ = Eˆ
( u / v ) , This value can be shown to be (see appendix):
j
j
j
2
(4) ~ σ u
u ( ( ˆ
j = Y.
) . )
2 2
j − Xβ
j ,
( σ + σ / n )
u
e
j
where β ˆ is the MLE of β for the aggregate model. Notice that this estimator has the same
shrinkage factor as the disaggregate estimator.
e
J
⎤
⎥
⎥
⎥
⎦

However, the school effects in (4) are heteroskedastic, while the true school effects are
not. Thus, to correct for heteroskedasticity, we divide the estimator by its standard deviation
obtaining the standardized estimator of school effect:
( 1
(5) u ( ( ˆ
j =
Y.
j − Xβ
) . j )
2 2
σ + σ / n
u
e
j
Thus, the set of u j
( ’s may also be used to rank schools. Similarly, the multilevel estimator in (2)
can also be standardized to obtain:
)
2 2
n j
(2.a) u j = ( 1/( σ u + σ e / n j ))( ∑ yˆ
i ij ) / n
= 1 j
1.2. Confidence Intervals for the Estimates of School Quality
A confidence interval for school effects is: uˆ j ± t1
/ 2σ
u|
uˆ
. Thus, it is necessary to obtain
the conditional variance of the random effect given its estimator; that is, Cov ( u | uˆ
) .
For both, disaggregate and aggregate estimators, the covariance matrix is derived
similarly. First it is necessary to obtain the joint distribution of the vector of school effects u and
its estimator. For this, notice that in both cases, the estimator is a linear combination of the vector
of dependent variables, test scores in our case. Thus the joint distribution can be derived from the
joint of u and Y. Then, using a theorem from Moser (theorem. 2.2.1, page 29), the conditional
covariance matrix of school effects is obtained. A derivation of this covariance matrix is given in
the appendix.
The conditional covariance matrix based on the disaggregate estimator is:
2 4 −1
−1
(6) Cov u | uˆ
) = I −σ
Ζ'
V V − X ( X'V
X )
9
−α
−1
−1
( X')
V Ζ
( σ u u
,
The conditional covariance matrix based on the aggregate estimator is:

2 4 −1
' −1
' −1
(7) Cov( u | u
~
) = σ u I −σ
uV
a ( Va
− X a (X aVa
X a ) X a ) Va
.
1.3. Bias in Estimation Introduced by Measurement Error
Let us consider a two-level model with measurement error. The model is:
(8) y ij = ( x β ) ij + u j + eij
, i = 1, K , n j j = 1,
K,
J
Y = y + q
ij
X hij = xhij
+ mhij
, h = 1,
K,
H
cov( q ij , qi'
j ) = cov( mhij
, mhi'
j ) = 0
E ( q ) E(
m ) = 0
ij
ij
= hij
cov( = σ
h ij h ij h h m
m m , ) , )
1
where yij is the real test score for the i th student in the j th school, qij is the measurement error for
2
y ij , ij ~ ( 0,
q ) N q σ , Yij is the observed test score, xhij is the true measure of the h th student or
school characteristic corresponding to the i th student in the j th school, mhij is the measurement
error for x hij , u j is the random component for school j, eij is the residual, and σ h , h ) m is the
covariance of measurement errors from two explanatory variables, h1 and h2, for the same
2
student. The covariance of measurement errors from any two variables is assumed to be equal for
all students regardless of the school they attend.
10
ij
( 1 2
( 1 2
Following Goldstein (1995), without measurement error, the parameters β could be
estimated by the FGLS estimator ˆ ˆ − 1 −1
( x V x)
( x Vˆ
−1
β = ′ ′ y)
. But measurement error as defined
− 1 −1
−1
−1
by model (8) implies that E ( x′
V x)
= ( X ′ V X ) − E(
m′
V m)
; so an unbiased estimator for
β in the presence of measurement error is proposed by Goldstein (1995) to be:

(9) ˆ − 1 ˆ −1
−1
[ X V X ( m V m)]
( X Vˆ
−1
β = ′ − E ′
′ Y )
)
.
When measurement error is not taken into account, the matrix ( m V m)
1 −
E ′ is omitted.
Using Goldstein’s derivation of ( m V m)
1 −
E ′ and realizing that the inverse of V is also a block
2 2
( n j −1)
σ u + σ e
diagonal with elements
in the diagonal, each element ( 2 2 2
1, 2 )
σ ( n σ + σ )
h h of the H × H
matrix ( m V m)
1 −
E ′ can be expressed as
e
j
2
J ⎧ σ ⎫ σ
u
( h1
, h2
) m
(10) ∑ n
= ⎨ − +
j j ( n
1 j 1)
1⎬
.
2
2 2
⎩ σ e ⎭ n jσ
u + σ e
u
e
Now let us see how this omitted matrix, ( m V m)
1 −
E ′ , compares with the one obtained when
aggregating the model. Aggregating the true disaggregate model, we obtain:
(11) = x β ) + u + e , j = 1,
K,
J
where notation is as in model (8).
y. j ( . j j . j
Y . j = y.
j + q.
j
X h.
j = xh.
j + mh.
j
cov( q . j , q.
j'
) = 0
E ( q.
j ) = E(
mh.
j ) = 0
m m = σ / n
cov( h ij , h ij ) h , h ) m
1
2
11
( 1 2
Notice how the covariance of measurement error between any two fixed explanatory
variables is reduced in the aggregate model. Now the covariance matrix of the true model is a
diagonal matrix with elements defined in the first part of this section; and which will be denoted
j

y V a . Following a procedure analogous to Goldstein’s derivation for the disaggregate model,
one can obtain the following unbiased estimator of β for the aggregate model:
(12) ˆ ˆ −1
( ˆ −1
−1
[ X V X m V m )] ( X Vˆ
−1
β = ′ − ′
′ ) ,
a a a a E a a a a a Ya
where the subscript a denotes aggregate data. As can be seen, the bias now will depend on
−1
E( m′
aVa
m a ) , an H × H matrix whose ( 1, 2 ) h h element is
J σ ( h1
, h2
) m
(13) ∑ .
2 2
= n σ + σ
j 1 j u e
As can be seen by comparing values in (10) and (13), the bias in β due to measurement
error is attenuated in the aggregate model. Bias in the estimation of β without accounting for
measurement error, is likely to affect the estimators of school effects, as suggested in (2) and (4).
This result is worth considering since adjustments for measurement error are seldom made and,
as Woodhouse et al. (1996) argue, different assumptions about variances and covariances of
measurement error may lead to totally different conclusions (when ranking schools, for
example). Therefore, when not correcting for measurement error, gains from aggregation may
somewhat offset the negative consequences of aggregation. Then, at least asymptotically,
aggregate estimates of school effects may be less inaccurate than what researchers have claimed.
However, to examine the properties of our aggregate and disaggregate estimators of
school effects in small samples, a Monte Carlo study will be necessary. Also, from the study we
will be able to compare the estimators’ asymptotic and small sample behavior.
12

2. Data and Procedures
A Monte Carlo study was used to compare aggregate and disaggregate estimates of
school effects with their true values. These values were also compared to OLS estimates with
aggregate data since this is what is most often done. The model on which the data generating
process was based, was taken from Goldstein’s 1997 paper, table 3, page 387, because it was
simple, and provided estimates of the random components for school and student, based on real
data.
This model regresses test scores of each student against a previous test score, a dummy
variable for gender, and a dummy for type of school (boys’, girls’, or mixed school). Test scores
were transformed from ranks to standard normal deviates. The random part consists of the school
effect and the student effect.
According to Goldstein, multilevel analysis provides the following estimated model:
Tˆ score = -0.09 + 0.
52Pscore
+ 0.
14Girl
+ 0.
10GirlsSch
+ 0.
09BoysSch
,
ij
(14) i = 1 , K , n j j = 1,
K J .
ˆ 2 =
u
ij
ij
The estimated variance of school effects, also called between-school variance, is
σ 0.
07 , and the variance of student effects, also called within-school variance, is ˆ 0.
56
2 σ = .
These values and the estimates of the fixed part of the model were used to generate the
disaggregate data. At each replication, n j observations were generated for each school, where n j
was a random realization of a lognormal distribution. Lagged test scores were generated from a
standard normal. Dummy variables were generated from binomial distributions. The random
components of the model for school and student were generated using a normal with zero mean
and variance ˆ 0.
07
2 σ = and ˆ 0.
56
2 σ = respectively, and the actual test score was obtained as in
u
e
13
j
j
e

equation (2). Then measurement error was introduced to the previous and actual test scores.
Measurement error was assumed to be a normal random variable with a zero mean and a
standard deviation of 0.3. All dummy variables are assumed measured without error.
Once a disaggregate data set is generated, estimates for school effects and variance
components are obtained using multilevel analysis as provided by the Mixed procedure in SAS.
Then, the disaggregate data set is aggregated by schools. Residuals as well as the two
components of the variance of the error term are estimated using NLMIXED in SAS. At this
point, we will have a set of 100 true school effects (since the number of schools in the sample is
100), and two sets of estimated school effects using aggregate and disaggregate data. Each set is
used to rank the schools in the sample. The greater the school effect, the better the school’s
performance, and therefore, the higher its ranking. We also provide rankings with standardized
school effects and the OLS estimate of school effects. Finally, we compute the estimated
variance components under all approaches and compare them with the true values.
A comparison of the school effect estimators is done in several different ways. Estimated
magnitudes of school effects are compared to the true magnitudes with the root mean squared
error (RMSE). This statistic measures the sum of squared deviations of estimated versus true
school effects, so the smaller the RMSE, the better the performance of the estimate. Spearman’s
correlation coefficient is calculated for all estimators in order to measure the degree of
correlation of each ranking with the true schools ranking. Finally we compare the top-ten set of
schools obtained with each estimator, with the true top-ten set 1 . The whole process described
above constitutes a single replication of the Monte Carlo study. As many as 1000 replications
were used.
14

Outcomes with and without measurement error are compared in order to see if the
aggregate estimator is in fact more robust to errors in measurement than the disaggregate
estimator. The parameters used to randomly generate the number of students in each school are
also changed, to see how variability in school size affects the performance of the estimators.
3. Results
Table 1 shows the first set of results for 1000 samples, each of 100 schools whose size is
distributed lognormal with mean 120 and variance 50000. According to this distribution, about
70% of schools have sizes between 15 and 250 students. As expected, the disaggregate estimator
performs best on almost all measures. The aggregate estimator’s performance, however, is very
good, and clearly above the OLS estimator’s performance. OLS tends to pick small schools as
the top schools. The average school size for the top ten schools as estimated by OLS is about
102, while the true average for this group is 120. OLS estimators are based on residuals whose
2 2
variance is σ + σ / n . So, quality estimates for small schools will have a larger variance and
u
e
will be more likely to be either at the bottom or top of the rankings. However, table 1 shows that
both the aggregate and disaggregate estimators tend to pick large schools as the top schools so
they are also a biased predictor of top schools.
The aggregate and disaggregate estimators have a shrinkage factor reduces the residuals
of small schools. Recall the shrinkage factor is
2
σ u
2 2
σ u + σ e / n
15
. This factor is always less than one, but
decreases with school size, bringing down the absolute value of small school residuals. Results in
table 1 suggest that the shrinkage factor may over-compensate for the residuals effect, and thus,
leave mainly large schools in the extremes. Estimators with a smaller shrinkage factor (the factor

is
1
2 2
σ u + σ e / n
) such as the standardized aggregate (equation 5) and standardized disaggregate
estimators alleviate this problem. Table 1 shows how the average size for the top ten schools
according to the standardized estimators only differs by one student from the true top-ten group
size average.
When measuring the RMSE of the estimators with respect to the true magnitude of school
effects, we find again that the disaggregate estimator performs only slightly better than the
aggregate estimator. Of course, the standardized estimators are not meant to match the
magnitudes of school effects, so their RMSE’s are high and should not be compared to the non-
standardized versions. When measuring the performance of the estimators by their ability to
match the true ranking and not the true values of the school effects, the RMSE might not be as
good of a measure as all the others presented in the table. However, when magnitudes are
important, the non-standardized versions of these estimators should be used.
The between- and within-school variance estimates are presented in Table 1. Although
the aggregate point estimates are close to the true variances, by looking at the standard deviations
of these estimates, it is clear that aggregation reduces the ability to estimate the within schools
variance as compared to the disaggregate estimator. In fact, being able to estimate these variance
components is crucial to the performance of the aggregate estimator. The ability to estimate the
variance components is determined by the sample variation in school size. For the same mean of
120 students and a variance of 10000 (5 times smaller), 91% of schools would have sizes
between 15 and 250, and less than 1% would be smaller. In this case, it is almost impossible to
estimate the variance components and OLS performs better than the aggregate estimator.
Table 2 introduces measurement error that is 30% of the highest possible test score. We
had hypothesized that measurement error would have less effect on the aggregate estimators. Our
16

esults validate this hypothesis. However, aggregate data are more likely to suffer from errors in
measurement than disaggregate data. As stated in the introduction, this is due to student mobility,
and in general, the fact that averages are not taken over the same group of students.
Table 3 shows the results for samples with mean school size of 20 and a variance of 250,
which implies that 70% of schools will have sizes between 10 and 50 students. This is done to
consider the case when policy makers require evaluations at the grade rather than school level.
Results are as before; the aggregate estimator is better than OLS and only slightly worse than the
disaggregate estimator.
2
As school size increases, the variation in averaged residuals due to students ( σ e / n )
becomes insignificant and the averages come closer to their true means. This implies that
aggregation becomes less of a concern for estimating school effects and heteroskedasticity is
almost insignificant. The problem with small or large schools being consistently rewarded almost
disappears. In fact, table 4 shows results for a mean school size of 300 and variance of 100,000.
Differences among ranking measures have narrowed for all estimators, and OLS, the only
estimator that does not rely on estimating variance components performs at its best.
4. Conclusions
Researchers argue that value-added multilevel models provide the most accurate
measures of school quality. But most states continue to use aggregate data (usually not in a value
added framework) to rank and reward schools. Research criticizing aggregate models, by
comparing them with disaggregate models, have used ordinary least squares rather than
maximum likelihood estimators. This article shows that the criticisms of aggregate models have
been overstated.
17

Results show that when many small schools are present in the data, the proposed
aggregate data estimator performs better than OLS on aggregate data, and only slightly worse
than the disaggregate data estimator. However, as school size increases, the three estimates
perform more alike. Even though the aggregate data estimator is only slightly worse than the
disaggregate data estimator for ranking schools based on efficiency, we still want to encourage
the collection of disaggregate data because of their ability to handle differential effects and at
least partly address student mobility.
Reward systems based on OLS estimators tend to reward small schools over bigger ones,
as the empirical literature has shown, while the shrinkage disaggregate estimator rewards large
schools. A standardized version of this estimate is presented that eliminates this problem. Thus,
when school officials are able to collect multilevel data, this study suggests they should
standardize the estimates of school quality before ranking schools. However, when disaggregate
data are not available, and small schools are present in the sample the standardized aggregate
estimator proposed here should be used. Note that our application is to schools, but the results
are applicable to measuring efficiency in any industry where aggregate data may be the only data
available.
18

Notes
1 Although the aggregate estimate is theoretically unbiased, a test for bias similar to Hanushek
and Taylor’s was performed. Hanushek and Taylor find that aggregation biases downward the
estimated school effects. They reestimate the value-added equation entering an estimate of
school quality as a fixed effect. Bias of the school effect estimate is measured by deviations of
the coefficient of school quality from one. In our Monte Carlo study, rather than an estimate, the
true school effects can be used, and therefore, a generated regressor problem is avoided. No
evidence of bias is found and thus Hanushek and Taylor’s finding of bias is apparently due to a
bias in the construction of their test rather than due to aggregation.
19

References
Clotfelter, C. T., & Ladd, H. F. (1996) Recognizing and Rewarding Success in Public Schools in
H. F. Ladd (Ed.), Holding Schools Accountable. Performance-Based Reform in Education
Washington, D.C.: The Brookings Institute.
Dickens, W. (1990). Error Components in Grouped Data: Is It Ever Worth Weighting? Review
of Economics and Statistics, 328-333.
Fowler-Finn, T. (2001). Student Stability vs. Mobility. Factors that Contribute to Achievement
Gaps. School Administrator. August 2001. Available at
http://www.aasa.org/publications/sa/2001_08/fowler-finn.htm.
Goldstein, H. (1995). Multilevel Statistical Models. London: Edward Arnold.
Goldstein H. (1997). Methods in School Effectiveness Research. School Effectiveness and
School Improvement, 8, 369-395.
Hanushek, E. A., Rivkin S. G. & Taylor L. L. (1990). Alternative Assessments of the
Performance of Schools:Measurement of State Variations in Achievment. The Journal of Human
Resources, 25, 179-201.
Hanushek, E. A., & Taylor L. L. (1996). Aggregation and the Estimated Effects of School
Resources. The Review of Economics and Statistics, 2, 611-627.
20

Ladd, H. F. (1996). Catalysts for Learning. Recognition and Reward Programs in the Public
Schools. Brookings Review, 3, 14-17.
Moser, B. K. (1996) Linear Models: A Mean Model Approach. California: Academic Press.
Odden, A., & Clune W. (1995) Improving Educational Productivity and School Finance.
Educational Researcher, 9, 6-10.
Webster, W. J., Mendro, R. L., Orsak, T. H., & Weerasinghe, D. (1996) The Applicability of
Selected Regression and Hierarchical Linear Models to the Estimation of School and Teacher
Effects. Paper presented at the annual meeting of the National Council of Measurement in
Education, New York, NY.
Woodhouse, G., & Goldstein, H. (1998). Educational Performance Indicators and LEA League
Tables. Oxford Review of Education, 14, 301-320.
Woodhouse, G., Yang, M., Goldstein, H., & Rasbash, J. (1996) Adjusting for Measurement Error
in Multilevel Analysis. Journal of the Royal Statistical Society A, 159, 201-212.
21

Table 1. Estimates of school quality using aggregate vs. disaggregate data with no measurement
error.
Measure Type of estimator Mean Std.Dev.
Spearman Disaggregate 0.8852 0.0296
L:\Home\Brorsen\PAPERS\Oldpapers Feb 05\Measuring School Quality\aggdis1revised.doc 2/9/2006
Std. disaggregate 0.8803 0.0317
Aggregate 0.8724 0.0327
Std. aggregate 0.8700 0.0342
OLS 0.8603 0.0377
RMSE Disaggregate 0.1182 0.0130
Std. disaggregate 0.4714 0.0538
Aggregate 0.1294 0.0167
Std. aggregate 0.4950 0.0562
OLS 0.1516 0.0219
Top Ten Disaggregate 6.97 1.203
Std. disaggregate 6.96 1.178
Aggregate 6.67 1.284
Std. aggregate 6.75 1.207
OLS 6.57 1.258
School Size Avg. Real Group 120.19 72.55
In Top Ten Group Disaggregate 140.88 80.91
Std. disaggregate 119.95 67.66
Aggregate 143.21 83.90
Std. aggregate 120.75 67.19

OLS 101.94 62.86
Variance Estimates Dis. Within Sch. 0.560 0.008
Dis. Between Sch. 0.070 0.012
Agg.Within Sch. 0.556 0.444
Agg Between Sch. 0.067 0.015
Note: Results are for 1000 simulations, each including 100 schools. The number of students per school is
a lognormal random variable with mean 120 and variance 50000. Mean is the average over all
simulations, RMSE is root mean squared error, Top Ten is the average number of schools ranked in the
top ten with the estimator, that belong to the true top ten set. Estimators compared are the disaggregate
estimator, its standardized version, the aggregate estimator, its standardized version, and the OLS
estimator of school effects. Variance estimates are also presented for the disaggregate and aggregate
methods.
1

Table 2. Estimates of school quality using aggregate vs. disaggregate data with measurement
error.
Measure Type of estimator Mean Std.Dev.
Spearman Disaggregate 0.8557 0.0321
Std. disaggregate 0.8503 0.0347
Aggregate 0.8581 0.0326
Std. aggregate 0.8544 0.0347
OLS 0.8423 0.0389
RMSE Disaggregate 0.1324 0.0130
Std. disaggregate 0.5283 0.0552
Aggregate 0.1364 0.0164
Std. aggregate 0.5248 0.0567
OLS 0.1667 0.0278
Top Ten Disaggregate 6.50 1.197
Std. disaggregate 6.45 1.188
Aggregate 6.48 1.235
Std. aggregate 6.53 1.185
OLS 6.29 1.188
School Size Avg. Real Group 116.36 64.52
In Top Ten Group Disaggregate 137.19 68.01
Std. disaggregate 116.00 64.28
Aggregate 141.90 73.20
Std. aggregate 117.93 65.74
2

OLS 95.92 60.25
Variance Estimates Dis. Within Sch. 0.670 0.009
Dis. Between Sch. 0.073 0.013
Agg.Within Sch. 0.654 0.484
Agg Between Sch. 0.067 0.016
Note: Results are for 1000 simulations, each including 100 schools. The number of students per school is
a lognormal random variable with mean 120 and variance 50000. Measurement error is 30% of highest
test score, in actual and previous scores. Mean is the average over all simulations, RMSE is root mean
squared error, Top Ten is the average number of schools ranked top ten with the estimator, that belong to
the true top ten set.
3

Table 3. Estimates of school quality using aggregate vs. disaggregate data for small schools.
Measure Type of estimator Mean Std.Dev.
Spearman Disaggregate 0.7684 0.0140
Std. disaggregate 0.7637 0.0140
Aggregate 0.7580 0.0178
Std. aggregate 0.7558 0.0168
OLS 0.7457 0.0185
RMSE Disaggregate 0.1643 0.0134
Std. disaggregate 0.6610 0.0570
Aggregate 0.1804 0.0227
Std. aggregate 0.6784 0.0593
OLS 0.2212 0.0227
Top Ten Disaggregate 5.47 1.302
Std. disaggregate 5.50 1.296
Aggregate 5.29 1.315
Std. aggregate 5.39 1.284
OLS 5.23 1.333
School Size Avg. Real Group 19.43 5.08
In Top Ten Group Disaggregate 23.12 5.65
Std. disaggregate 19.66 5.20
Aggregate 23.02 5.60
Std. aggregate 19.61 4.69
OLS 16.44 4.64
4

Variance Estimates Dis. Within Sch. 0.560 0.018
Dis. Between Sch. 0.070 0.015
Agg.Within Sch. 0.537 0.358
Agg Between Sch. 0.066 0.026
Note: Results are for 100 simulations, each including 100 schools. The number of students per school is a
lognormal random variable with mean 20 and variance 250. Mean is the average over all simulations,
RMSE is root mean squared error, Top Ten is the average number of schools ranked top ten with the
estimator, that belong to the true top ten set.
5

Table 4. Estimates of school quality using aggregate vs. disaggregate data for large schools.
Measure Type of estimator Mean Std.Dev.
Spearman Disaggregate 0.9560 0.0153
Std. disaggregate 0.9557 0.0155
Aggregate 0.9425 0.0198
Std. aggregate 0.9438 0.0195
OLS 0.9437 0.0197
RMSE Disaggregate 0.0744 0.0123
Std. disaggregate 0.2913 0.0480
Aggregate 0.0904 0.0200
Std. aggregate 0.3280 0.0529
OLS 0.0846 0.0139
Top Ten Disaggregate 8.08 1.020
Std. disaggregate 8.08 1.020
Aggregate 7.66 1.151
Std. aggregate 7.77 1.069
OLS 7.81 1.085
School Size Avg. Real Group 301.07 96.76
In Top Ten Group Disaggregate 313.28 101.81
Std. disaggregate 302.47 100.95
Aggregate 330.69 102.75
Std. aggregate 313.33 96.67
OLS 297.42 99.50
6

Variance Estimates Dis. Within Sch. 0.560 0.005
Dis. Between Sch. 0.698 0.011
Agg.Within Sch. 0.972 1.433
Agg Between Sch. 0.063 0.013
Note: Results are for 100 simulations, each including 100 schools. The number of students per school is a
lognormal random variable with mean 300 and variance 100000. Mean is the average over all
simulations, RMSE is root mean squared error, Top Ten is the average number of schools ranked top ten
with the estimator, that belong to the true top ten set.
7

Appendix
Derivation of the aggregate estimators of school effects
Recall equation (3.a) which shows the aggregate model:
L:\Home\Brorsen\PAPERS\Oldpapers Feb 05\Measuring School Quality\aggdis1revised.doc 2/9/2006
Y = X β + u + e
a
However, the aggregate model has no way of differentiating among its random terms, thus we
rewrite the model as:
a
Y = X β + w .
a
We are to obtain the conditional mean of u given the total residual w = u + e a based on the
distributions of u and e.
Since u and e are independent normal random vectors, its distribution is given by:
⎛u
⎞
⎜ ⎟ ~ N
⎝e
⎠
( 0,
V )
u, e
a
2 ⎡σ
⎤
u I J 0
, where Vu,e = ⎢
2 ⎥ , N being the total number of students.
⎣ 0 σ e I N ⎦
But ⎟ ⎛u
⎞
⎛u
⎞
⎜ is a linear combination of ⎜ ⎟ , this is:
⎝e
a ⎠
⎝e
⎠
⎛ I J
⎜
⎛u
⎞ ⎛u
⎞ ⎜
⎜
⎟ = A1
⎜ ⎟ = ⎜
⎝e
⎠ ⎝e
⎠
0
a
⎜
⎜
⎝
⎡
⎢
⎢
⎢
⎣
1
n
1
1′
distribution will be as follows:
0
n
1
0
O
1
n
J
0
1′
n
J
a
⎞
⎟
⎤⎟⎛
u⎞
⎥⎟⎜
⎟ , where 1 n is an nj vector of 1’s. Thus, its
j
⎥⎟⎝
e ⎠
⎥⎟
⎦⎠
⎛u
⎞
⎜
⎛
⎜
⎟ ~ N 0,
A1
V
⎝e
⎠ ⎝
a
u,e
′
A ⎟
⎞
1 .
⎠
⎛u
⎞
From this random vector, we construct ⎜ ⎟ pre-multiplying
⎝ w
⎟
⎠
⎟
⎛u
⎞
⎜ by
⎝e
⎟
a ⎠
⎟
⎛ I J 0 ⎞
A2 = ⎜ . Then, its
⎝ I J I J ⎠
distribution will be:

⎛u
⎞
⎜
⎛
′ ′
⎜ ⎟ ~ N 0,
A
⎟
⎞
2 A1
Vu,e
A1
A2
.
⎝ w ⎠ ⎝
⎠
Having the joint distribution of u and w = u + e a , our estimator is easily derived (Moser, theorem
2.2.1) as:
E(
u | w)
= Cov(
u,
w)
Cov(
w)
1
−1
⎡
⎢
( w)
= ⎢
⎢
⎢⎣
2
u
σ + σ / n
2
u
σ
σ
2
u
2
e
2
u
2
e
M
σ + σ / n
Derivation of the conditional covariance matrix Cov ( u / uˆ
)
Disaggregate data: Recall equation (1.a):
Y = Xβ
+ Ζ u + e
⎡1n1
⎢
Ζ = ⎢
⎢
⎣
0
O
Ζ u + e ~ N
1
0
n
J
( 0,V )
⎤
⎥
⎥
⎥
⎦
J
1
w ⎤
1
⎥
⎥
w ⎥
J ⎥⎦
The shrinkage estimator of school effects (equation 2) in matrix notation is:
2 −1
uˆ
σ Z'V
(Y X ˆ
−1
−1
−1
−1
= u − β ) or u = Z'V
(I − X ( X'V
X ) X'V
)Y
2 ˆ σ u
(*)
This shows clearly that the shrinkage estimator is a linear combination of the independent
variable vector.
Thus, we can derive the joint distribution of ( u, uˆ
)' by knowing the distribution of ( u, Y )' .
The distribution of ( u, Y )' is:
2
⎡u
⎤ ⎛ ⎡
⎜
⎡0
⎤ σ u I
⎢ ⎥ ~ N
⎜ ⎢ ⎥,
⎢ 2
⎣Y
⎦ ⎝ ⎣Xβ
⎦ ⎣σ
u Z
2
σ ⎤⎞
u Z'
⎥⎟
.
V ⎟
⎦⎠
In general, u and any linear combination of Y of the form u ˆ = AY , will be jointly distributed as
follows:

2
⎡u⎤
⎛ ⎡
⎜
⎡0
⎤ σ u I
⎢ ⎥ ~ N
⎜ ⎢ ⎥,
⎢ 2
⎣uˆ
⎦ ⎝ ⎣Xβ
⎦ ⎣σ
u AZ
2
2
σ ⎤⎞
u Z'
A'
⎥⎟
AVA' ⎟
⎦⎠
Then, by Moser’s theorem 2.2.1, the conditional covariance is:
2 4
−1
Cov( u | uˆ
) = σ u I −σ
u Z' A' (AVA' ) AZ .
2 −1
−1
−1
Equation (6) is obtained by replacing A with σ Z'V
(I − X ( X'V
X ) X'V
) , from (*), in
the expression above.
Aggregate data: Again, we will use the same argument. First, re-express the aggregate
estimators of school quality in matrix notation:
~ 2
u
u = σ V
−1
a
(Y
a
− X ˆ β )
The distribution of ( u, Ya
)' is:
a
u
−1
−1
−1
−1
u V (I X X 'V
X X 'V
)Y
~ σ (**)
, or = u a − a ( a a a ) a a a
2
2
⎡u
⎤ ⎛ ⎡
⎜
⎡0
⎤ σ u I
⎢ ⎥ ~ N
⎜ ⎢ ⎥,
⎢ 2
⎣Ya
⎦ ⎝ ⎣X
a β ⎦ ⎣σ
u I
2
σ ⎤⎞
u I
⎥⎟
V ⎟
⎦⎠
So, the distribution of u and AYa, a linear combination of Ya is:
⎡u⎤
⎢ ⎥
⎣u
~ ~
⎦
and the conditional covariance matrix is:
2
⎛
⎜
⎡0
⎤ ⎡σ
u I
⎜ ⎢ ⎥,
⎢
⎝ ⎣X
a β ⎦ ⎣σ
u A
N 2
2
σ ⎤⎞
u A'
⎥⎟
,
AV A' ⎟
a ⎦⎠
2 4
−1
Cov( u | u
~
) = σ u I − σ u A' (AVA' ) A .
−1
−1
−1
−1
= u a
a a a a a a
2
When A σ V (I − X ( X 'V
X ) X 'V
) , we obtain equation (7).
−1