27.04.2022 Views

2020-2021_LU_EvaluationReport

Create successful ePaper yourself

Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.

RESEARCH REPORT

BAKER EVALUATION RESEARCH CONSULTING October 2021

21 st Century Community Learning Center:

Lucille Umbarger Elementary School

YEAR 4 EVALUATION REPORT

STACY M MEHLBERG, PH.D.

RONAK PATEL, PH.D.


Duane Baker is the founder and president of Baker

Evaluation, Research, and Consulting, Inc (The BERC

Group). Dr. Baker has a broad spectrum of public school

educational and program experience, including serving as a

high school classroom teacher, high school assistant principal,

middle school principal, executive director for curriculum and

instruction, and assistant superintendent. In addition, he has

served as an adjunct instructor in the School of Education at

Seattle Pacific University since 1996, where his emphasis has

been Educational Measurement and Evaluation and

Classroom Assessment.

Dr. Baker also serves as the Director of Research for the

Washington School Research Center at Seattle Pacific

University. He also serves as an evaluator for several

organizations including the Bill & Melinda Gates Foundation,

Washington Education Foundation, Washington State Office

of Superintendent of Public Instruction, and others.

Members of The BERC Group have K–20, experiences as

teachers, counselors, psychologists, building administrators,

district administrators, and college professors. The team is

currently working on research and evaluation projects at the

national, state, regional, district, school, classroom, and

student levels in over 1000 schools in Washington State and

nationally.

COPYRIGHT © 2020 BY THE BERC GROUP INC. ALL RIGHTS RESERVED

ADDITIONAL COPIES OF THIS REPORT MAY BE OBTAINED THROUGH THE BERC GROUP (www.bercgroup.com).

THE BERC GROUP

COPYRIGHT © 2011 BY THE BERC GROUP LLC. ALL RIGHTS RESERVED

ADDITIONAL COPIES OF THIS REPORT MAY BE OBTAINED THROUGH THE

BERC GROUP (www.bercgroup.com).


Table of Contents

EXECUTIVE SUMMARY

21st Century Community Learning Center .....................................1

Lucille Umbarger Elementary School .............................................1

Introduction .......................................................................................1

Community Context .........................................................................................1

Grant Performance Objectives .......................................................................2

Evaluation Design .............................................................................2

Evaluation Questions ......................................................................................2

Data Sources ....................................................................................................3

Implementation Questions ...............................................................3

Q1. To what extent were components of the grant implemented with

fidelity during Year 4?......................................................................................4

Q2. What challenges emerged during Year 4 implementation? ..................5

Q3: What promising practices emerged during Year 4 implementation?...5

Impact Questions ..............................................................................6

Q4: 4. To what extent does program participation relate to improved

academic and social outcomes? ....................................................................6

Q5. To what extent are program components sustainable and

continuously improving? ................................................................................7

Youth Program Quality Assessment ...............................................7

Conclusions and Recommendations ............................................10

References.......................................................................................12

THE BERC GROUP


21 st Century Community Learning Center

Lucille Umbarger Elementary School

YEAR 4 EVALUATION REPORT

Introduction

The purpose of this report is to provide formative and summative feedback to personnel

at the Boys & Girls Club of Skagit County (BGCSC) regarding the 21 st Century

Community Learning Center (21 st CCLC) at Lucille Umbarger. This was the fourth year

of programming at this sight.

Community Context

Lucille Umbarger Elementary School served approximately 576 students in kindergarten

through eighth grades during the 2020-2021 school year. The school is located within the

Burlington-Edison School District in Burlington, Washington. Of the total school

population, over 70% identify as low income, and approximately one quarter of the

students are English Language learners. OSPI Report Card shows that for the 2018-2019

school year 1, 28% of students met ELA grade level standards, and 24% met math Grade

level standards. Community demographics are presented in Table 1.

Table 1. Lucille Umbarger Community Demographics

Federal Race/Ethnicity 2017-2018 2018-2019 2019-2020 2020-202

American Indian/Alaskan Native 1.2% 1.6% 1.3% 1.4%

1

Asian 2.1% 1.7% 1.3%

Hispanic/Latino of Any Race 55.0% 55.2% 55.6% 60.1%

Two or More Races 4.7% 4.4% 5.7% 5.4%

White 37.0% 37.1% 36.1% 31.4%

Characteristic

Low Income 69.1% 69.8% 72.8% 71.9%

English Language Learner 29.0% 28.6% 26.8% 25.3%

1 This was the most recent administration of state standardized tests. This information will be updated

when new scores are available.

THE BERC GROUP

1


Migrant 3.0% 2.5% 4.1% 4.7%

Students with Disabilities 14.0% 15.6% 15.9% 15.8%

Grant Performance Objectives

The 21st CCLC at LU continued building their program, while also adjusting to the

expectations and challenges created by the COVID pandemic. The program director

shared, “I think this year the kids just really needed to connect with one another. The kids

were fairly individual this year, with school being remote. They have missed out on the

social connections.” In addition to this targeted attention to the social-emotional needs of

students, program staff continued to focus on the grant objectives.

1. Improve student academic achievement via literacy and STEM instruction,

regular homework help and subject-specific tutoring.

2. Increase student interest in school, especially in English, Math, and Science

through project-based learning and relevant service learning

3. Improve student self-efficacy, sense of agency, social-emotional learning and

school engagement

4. Increase sense of self and personal identity through place-based development of

cultural awareness and environmental literacy

5. Increase family literacy, bilingualism, and educational aspirations.

Evaluation Design

Researchers used a multiple measures, mixed methodology approach to conduct this

evaluation. The collection of both quantitative and qualitative data adds scope and

breadth to the study in addition to providing the ability to triangulate findings (Creswell,

1994). Additionally, researchers followed a utilization-focused model of evaluation. This

model of evaluation is intended to provide useful feedback that can be implemented in

real time (Patton, 2013). Qualitative data was gathered during program observations,

interviews with staff and document review. The BGCSC provided a database of student

attendance in programming. State testing was not administered during the 2019-2020 or

2020-2021 school years due to the ongoing pandemic.

Evaluation Questions

The evaluation questions were designed to provide program leaders formative feedback

for continuous improvement, as well as summative feedback about progress toward

2

THE BERC GROUP


BGCSC 21 st Century After-School Program grant goals. The questions align with OSPI’s

local evaluation guidelines for 21 st Century Community Learning Center programs and

specific performance objectives of the BGCSC. The evaluation questions are:

Implementation Questions (Focused on the effectiveness of plans, processes, and

program strategies related to the grant vision and goals):

1. To what extent were components of the grant implemented with fidelity?

2. What challenges emerged during Year 3 implementation?

3. What promising practices emerged during Year 3 implementation?

Outcome Questions (Focused on meeting local, state, and federal performance goals and

objectives):

4. To what extent does program participation relate to improved academic and social

outcomes?

5. To what extent are program components sustainable and continuously improving?

Data Sources

To address these questions researchers gathered data from multiple sources throughout

the evaluation cycle. BGCSC provided a database of student attendance 2. Additionally,

BERC researchers conducted:

• Collection and analysis of interviews in Spring 2021

• Analysis of After-school program student attendance data

• Collection and analysis of School Age Program Quality Assessment (SA-PQA)

• Observation of the program

• Initiative documents and materials review

Implementation Questions

The following evaluation questions focus on program implementation during Year 4,

including the effectiveness of plans, processes, and program strategies relating to the

grant vision and goals. These questions were primarily assessed through staff interviews,

observations, and ongoing correspondence with program stakeholders.

2 Achievement data will be included in future reports, once available.

THE BERC GROUP

3


Q1. To what extent were components of the grant implemented with fidelity during

Year 4?

(Focus on academic support, engagement, social emotional learning, family and

community partnerships and continuous improvement.)

During Year 4, the LU 21 st CCLC staff continued to focus on programming aligned with

the values and mission of the B&G Club, while also navigating the limitations placed

upon them as a result of the COVID pandemic. Program staff demonstrated a clear

commitment to serving students by providing high-quality academic and enrichment

opportunities, despite low attendance throughout the year. The program director shared,

“I am really impressed with the quality and structure of the programming. The staff have

it together. [They have a] community wall with the stars, the interactions are really

positive, and the staff is working through and resolving issues consistently with kids.

[They] have been really successful with SEL and behavioral interventions.”

When asked to discuss their academic programming, one staff member outlined their

typical process. She shared,

We ask kids to do homework first, then reading, then power pages. I have a folder

for each with grade appropriate content that targets the skills they need to work

on. Sometimes they have self-directed projects that they need to finish from class.

We have a power point system- there is a grid on the board of 250 power pointseach

assignment has a point value- and they can earn bonus points. They get to

initial one of the squares, and they can work together to earn a party. We used to

do it individually, but it has seemed to work better for this group of kids to work

together.

Staff members noted that they continued to reach out to teachers for student information

to better address individual needs. One staff member shared, “If a kid is falling behind, a

teacher might come to us and ask us if we can help them. [Our site coordinator] has been

good about talking with the staff, and has talked to the principal and front office. We are

talking about being more intentional, and thinking ahead for next year.”

In addition to academic support, the 21st CCLC staff provided enrichment opportunities

focused on social emotional learning and issues of diversity, equity, and inclusion.

Students participated in a 6-week curriculum from Youth for Unity, which taught students

about unconscious bias and white privilege. Staff also worked to incorporate SEL into

everything they did in an effort to build one another up and promote positive self-

4

THE BERC GROUP


awareness. Staff implemented The Community Corner, where they would post their

program values and give one another ‘shout outs’. One staff member shared, “[We are]

trying to build community. We have a lot of kids working on self- regulation and [setting]

goals for themselves. A lot of SEL just happens naturally, but [we] have pulled a little

from Sanford and Harmony.” The site coordinator noted, “We have a lot of success when

the staff finds things that are specific to their students. We are not just relying on those

curricula. We have a lot of resources, and are modifying them to meet the needs of the

kids in the moment.”

Q2. What challenges emerged during Year 4 implementation?

Program leadership and staff shared that most of the challenges during Year 4 were

directly related to the pandemic. COVID scheduling was difficult, as things would open,

then close, making it so the “window of opportunity did not always coincide with the

state and district regulations.” One staff member discussed their i-clubhouse program that

was implemented during the initial COVID school closures in 2019-2020, but noted that

they tried carrying the program into the fall, and it didn’t get traction once students were

learning from home. She felt “it was too much computer time, and families weren’t really

into it.” As a result of these challenges, student attendance was low. One staff member

noted, “One of the attendance issues is that we still had to operate after school, [and] it

has been much more difficult for families to get here after school if the kids weren’t in

school full day.”

In addition to Covid and attendance, staffing remained a persistent challenge for LU. The

site coordinator shared, “Here all of our part time staff are school district employees.

There are opportunities for district staff to grow through the school district, so we lose

staff sometimes to other opportunities.”

Q3: What promising practices emerged during Year 4 implementation?

The 21 st CCLC program at LU has had challenges over the past few years related to

program and staff stability. Despite these difficulties, program leaders and staff felt that

they had developed a strong program for the students who were able to attend, and

dedicated time and energy to providing the social, emotional and academic support their

students needed during the pandemic. LU 21 st CCLC staff were organized, dedicated to

building relationships, and successful at creating routines and rituals to help create an

engaging and enriching program.

.

THE BERC GROUP

5


Impact Questions

Q4: 4. To what extent does program participation relate to improved academic and

social outcomes?

During interviews staff were asked to share their perceptions of student outcomes during

Year 4. Overall, staff noted that students seemed to benefit most from opportunities to

engage with their peers again after so much time isolated from friends and non-family

adults. Although it was difficult for staff to say with certainty whether students made

educational gains during Year 4, they did feel confident that students had developed more

adaptive social-emotional skills, and were more able to build relationships and ask for

what they needed because of the individualized attention and support they received

throughout the year.

In addition to anecdotal evidence of student academic and social outcomes, researchers

analyzed attendance data for students participating in the 21 st CCLC. As a result of the

school closures, no state assessment scores or discipline data were available for analysis.

Attendance. One outcome measure for the 21 st Century program is student attendance.

Program leaders collected and shared program attendance for all participating students.

Students were counted if they attended at least one day of program. Students that were in

attendance for more than 30 days were labeled “21 st Century” students. During Year 4,

LU had 23 students register, and 10 students attend at least 30 days (Table 2). This was a

sharp drop compared to the truncated 2019-2020 school year and was likely attributed to

distance learning and pandemic concerns.

Table 2. Lucille Umbarger Elementary Attendance

2019-2020 2020-2021

Number of Participating Students 60 23

Number of 21st Century Students 23 10

Percentage of 21st Century Students 38% 44%

When disaggregated by month, the total attendance was steady throughout the year,

between 8 and 11 students for most months (Figure 1).

6

THE BERC GROUP


Figure 1

Q5. To what extent are program components sustainable and continuously

improving?

Evaluators will continue to gather and track outcomes for grant objectives over time.

Although this report summarizes program efforts in Year 4, this site has had changes and

challenges that have impacted the timeline of progress, making longitudinal data more

difficult to track. Despite these challenges, the program staff and operating organization

are committed to serving the students, and have made significant changes intended to

strengthen the program and build a stronger connection to the school district and

community. In addition to the efforts to build a foundation for this 21 st CCLC, staff and

leadership also engaged in professional development and continuous improvement

efforts, including the Youth Program Quality Assessment.

Youth Program Quality Assessment

To fulfill grant requirements, researchers used the YPQA (Youth Program Quality

Assessment) tool to evaluate the quality of youth experiences in the program. This tool

assesses four main domains of programming, including safe environment, supportive

environment, interaction, and engagement. Youth-centered policies and practices, high

THE BERC GROUP

7


expectations for youth and staff, and access were other areas incorporated into the tool.

The YPQA was developed by the Weikart Center, and according to their website (2014),

“each domain contains items that focus on specific elements of best practice.” The

findings provide opportunities for staff members to reflect on what they are doing well, to

determine ways they can improve the program, and “to build professional competencies.”

Although the YPQA was not required during the 2020-2021 school year, results from

Year 3 are included to support the LU 21 st CCLC staff as they develop goals for their

future.

The overall Domain scores for Interaction and Engagement, which measure opportunities

for students to collaborate, lead, plan, and reflect, were low for the self and external

assessments, while Safe Environment scored high during both assessments (Figure 3).

4.6

4.6

LU YPQA Domain Scores

3.7

3.5

2.7

2.5

2

2

Safe Environment Supportive Environment Interaction Engagement

Figure 3

Self-Assessment

External Assessment

When disaggregated into individual scales, opportunities for leadership within the

Interaction domain scored at 1.67 out of 5, while collaboration, belonging and adult

partnerships scored higher (Figure 4). Within the Engagement domain, planning and

reflection were the lowest scoring scales (Figure 5).

8

THE BERC GROUP


LU YPQA Interaction Scale Scores

3

3 3

3

2.3

1.7 1.7

3

Belonging Collaboration Leadership Adult Partners

Figure 4

Self-Assessment

External Assessment

LU YPQA Engagement Scale Scores

3

3.5

1

1

2

1.5

Figure 5

Planning Choice Reflection

Self-Assessment

External Assessment

Opportunities for growth identified by self and external assessments include developing

student leadership capacity, building cooperation, and incorporating reflection into

learning activities. The Center for Youth Program Quality provided the following

recommendations to support further program growth:

o

Cooperative Learning: Providing young people an opportunity to participate in

and lead small groups has a positive impact on classroom climate, self-esteem

among students, internal locus of control, and time on task. Students in

THE BERC GROUP

9


cooperative teams are more active, self-directing, and expressive, all of which

may be associated with achievement gains.

o

Youth Voice: Providing young people with chances to make decisions about their

activities and how they carry them out can improve motivation and buy-in, and

more importantly, offering choices in a youth program space gives youth a chance

to practice for the bigger choices they'll make outside of the program.

o

Planning and Reflection: The skills of making plans for the future and learning

from the past can help youth succeed in school and in life. These skills are tied

into what brain scientists call executive functions, and play an important role in

directing attention to tasks and decision making that connects with consequences.

Conclusions and Recommendations

Throughout Year 4, the LU 21 st CCLC made progress toward many of the grant goals and

objectives. To continue to make improvements, we suggest the following

recommendations:

Collect and Use Data

Program leaders at each site should continue to document and collect data required for

the program and the evaluation. Additional training on the YQPA, as well as regular time

to perform and reflect on the tool should be allocated. This data provides critical, ongoing

information on student engagement and the effectiveness of enrichment activities, which

can be used to inform and improve the LU 21CCLC program. Additionally, the use of

real time student data will help program staff to help students to track assignments and to

be academically prepared. The sharing of data with the school, family and outer

community members may help to create a strong system of support, awareness, and

recognition for the program.

Strengthen Community and Family Partnerships

The LU 21 st CCLC program would benefit from increased community involvement and

support. By strengthening community partnerships, students engage in new experiences

and develop a broader world view. New program leaders may want to collaborate with

previous grant staff to gain insight about partnerships that are already established while

actively working to reach out to form new community collaborations.

10

THE BERC GROUP


THE BERC GROUP

11


Prepare for program growth and think about long term sustainable plans

This year, the program grew into a five day-a-week program serving expanded grades.

This growth created some challenges associated with funding, the availability of

resources, and staffing. Program leaders will want to plan for future program attendance

numbers and ensure the budget and staff ratio can appropriately accommodate the

growing needs of the program. As the program continues, staff members will also want to

think about ways they can sustain efforts beyond the availability of grant money.

Increase student choice, voice, cooperative learning, and reflection in projects

Recommendations from the YPQA suggest that site staff should increase opportunities

for student planning and incorporate student reflection into regular programming.

According to the YPQA report, “The skills of making plans for the future and learning

from the past can help youth succeed in school and life.” Ways to increase student

reflection include providing a dedicated reflection time at the start or end of the day,

asking three reflection questions at the end of each session, and by incorporating

journaling, partner talk, and exit slips. Staff and students alike may benefit from learning

strategies about how to ask reflective questions and about how to answer such questions.

Students may feel more invested in the program when they feel listened to and are an

active part of program planning. According to the YPQA, providing youth the chance to

make decisions about their activities and how they carry them out “can improve

motivation and buy in. Offering choices in the solace of a youth program space gives

youth a chance to practice for the bigger choices they’ll make outside of the program.”

Students may feel more invested in the program if they are regularly asked to share their

insights about projects and are encouraged to plan future activities. Lastly, it may

behoove staff to work on ways to incorporate opportunities for students to lead and

participate in small group learning. “Cooperative learning promotes student voice, allows

student to take on leadership roles, and “has a positive impact on classroom climate, selfesteem

among students, internal locus of control, and time on task” (YPQA, 2014).

References

David P.Weikart Center for Youth Program Quality. Youth Program Quality Assessment

and School-Age Program Quality Assessment. Obtained 8/27/2014 from http://

www.cypq.org/assessment

12

THE BERC GROUP


THE BERC GROUP

13


The BERC Group, Inc.

22232 - 17 th Ave. SE Suite 305

Bothell, WA 98021

Phone: 425.486.3100

Web: www.bercgroup.com

14

THE BERC GROUP

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!