Full Report - Center for Collaborative Education

A discussion of our HLM analyses of MCAS scores
and student–level characteristics and school environmental
factors is presented in ‘In Depth: Using
Hierarchical Linear Modeling (HLM) To Determine
the Relative Importance of Individual and School
Level Factors in LEP Students’ ELA and Math MCAS
Outcomes’ (see Chapter VIII). This appendix supplements
that discussion by providing additional information
from existing literature and by presenting
the results of the HLM analyses in more detail.
Existing Literature
Using HLM to analyze educational outcomes for
ELL students is a common approach in existing
research. The rationale for using HLM to study
outcomes for ELLs is the range in approaches to
ELL and LEP programs from school to school and
district to district. Even within the HLM research on
LEP students, there are several different approaches.
The most common approach is evaluating
student outcomes in the context of student-level
and school-level variables, including ELL/LEP placement
as a student-level covariate (e.g. Callahan,
Wilkinson, & Muller, 2010; Brown et al., 2010;
Wang, Niemi, & Wang, 2007).
While the HLM research on ELL students is far
from exhaustive, there are several factors that have
emerged as significant when analyzing educational
outcomes for these students. The literature using a
two-level linear model including student and school
level factors highlights the following significant
student level variables which were also found to be
significant in our study: gender (Brown, Nguyen,
and Stephenson, 2010; Rumberger and Thomas,
2000; Callahan, Wilkinson, & Muller, 2010; Wang
et al., 2007); language proficiency (Dawson & Williams,
2008; Wang et al., 2007, Hao & Bonstead-
Bruns, 1998); and being designated as a student
with disabilities (Wang et al., 2007). Attendance,
a behavioral variable, was also been found to be
significant (Rumberger, 1995; Rumberger & Palardy,
2005; Rumberger & Thomson, 2000). All of these
factors were considered in developing the HLM
models for this analysis. The literature typically
treats program participation as an individual level
variable and most frequently compares between
two different types of ELL programs (SEI, TBE,
2-way) or two different intensities of treatment
(ESL and ELL program). In this study we compared
the educational attainment of LEP students in ELL
programs with that of LEP students in general
education.
The literature also identifies several school level
variables that are consistently statistically significant
in two-level linear models. In particular, existing
literature highlights the following significant
school-level variables that were also found to be
significant in our study: school size (Werblow &
Duesbery, 2009; Wang et al. 2007; Rumberger
& Palardy, 2005; Lee & Smith, 1999; Lee & Bryk,
1989), school poverty level (Werblow & Duesbery,
2009; Braun et al, 2006; Lee & Smith, 1999, Hao
& Bonstead-Bruns, 1998), LEP density (Werblow &
Duesbery, 2009), and proportion of mobile students
(Rumberger & Palardy, 2005; Rumberger & Thomas,
2000). School quality variables are also mentioned
in the literature and found significant in our study,
such as the percentage of teachers that are highly
qualified/percentage of teachers that are licensed
in their subject (Munoz & Chang, 2008; Braun et
al. 2006, Rumberger & Palardy, 2005; Rumberger
& Thomas, 2000). In addition, we have included a
school’s AYP status in Math or ELA.
Results
The results of the HLM analyses support the findings
of the descriptive analysis presented in this
report. The key findings of the HLM analyses are
presented in the in-depth section; the following
tables present the detailed results of the HLM
analysis in each subject area (for more information
on variables and model development, please see
Appendix 1: Methods).
In the following tables, the plus and minus signs
represents positive (+) and negative (-) relationships
between the variables and the student’s MCAS
score. In other words, when the relationship between
the independent variable and MCAS scores
is positive, students’ MCAS scores tend to increase
as the variable increases; when the relationship is
negative, students’ MCAS scores tend to decrease
as the variable decreases. For the two-category
variables gender, SPED, program enrollment, and
AYP, a plus sign (+) indicates that the state of the
category indicated in the independent variable
list (e.g. ‘Female’) is associated with higher MCAS
scores, while a minus sign (-) indicates that the
other variable category (e.g. ‘Male’) is associated
with higher scores. Finally, the p-value indicates
whether or not the direction of the relationship is
Improving Educational Outcomes of English Language Learners in Schools and Programs in Boston Public Schools 137