2020-2021_LU_EvaluationReport

RESEARCH REPORT
BAKER EVALUATION RESEARCH CONSULTING October 2021
21 st Century Community Learning Center:
Lucille Umbarger Elementary School
YEAR 4 EVALUATION REPORT
STACY M MEHLBERG, PH.D.
RONAK PATEL, PH.D.

Duane Baker is the founder and president of Baker
Evaluation, Research, and Consulting, Inc (The BERC
Group). Dr. Baker has a broad spectrum of public school
educational and program experience, including serving as a
high school classroom teacher, high school assistant principal,
middle school principal, executive director for curriculum and
instruction, and assistant superintendent. In addition, he has
served as an adjunct instructor in the School of Education at
Seattle Pacific University since 1996, where his emphasis has
been Educational Measurement and Evaluation and
Classroom Assessment.
Dr. Baker also serves as the Director of Research for the
Washington School Research Center at Seattle Pacific
University. He also serves as an evaluator for several
organizations including the Bill & Melinda Gates Foundation,
Washington Education Foundation, Washington State Office
of Superintendent of Public Instruction, and others.
Members of The BERC Group have K–20, experiences as
teachers, counselors, psychologists, building administrators,
district administrators, and college professors. The team is
currently working on research and evaluation projects at the
national, state, regional, district, school, classroom, and
student levels in over 1000 schools in Washington State and
nationally.
COPYRIGHT © 2020 BY THE BERC GROUP INC. ALL RIGHTS RESERVED
ADDITIONAL COPIES OF THIS REPORT MAY BE OBTAINED THROUGH THE BERC GROUP (www.bercgroup.com).
THE BERC GROUP
COPYRIGHT © 2011 BY THE BERC GROUP LLC. ALL RIGHTS RESERVED
ADDITIONAL COPIES OF THIS REPORT MAY BE OBTAINED THROUGH THE
BERC GROUP (www.bercgroup.com).

Table of Contents
EXECUTIVE SUMMARY
21st Century Community Learning Center .....................................1
Lucille Umbarger Elementary School .............................................1
Introduction .......................................................................................1
Community Context .........................................................................................1
Grant Performance Objectives .......................................................................2
Evaluation Design .............................................................................2
Evaluation Questions ......................................................................................2
Data Sources ....................................................................................................3
Implementation Questions ...............................................................3
Q1. To what extent were components of the grant implemented with
fidelity during Year 4?......................................................................................4
Q2. What challenges emerged during Year 4 implementation? ..................5
Q3: What promising practices emerged during Year 4 implementation?...5
Impact Questions ..............................................................................6
Q4: 4. To what extent does program participation relate to improved
academic and social outcomes? ....................................................................6
Q5. To what extent are program components sustainable and
continuously improving? ................................................................................7
Youth Program Quality Assessment ...............................................7
Conclusions and Recommendations ............................................10
References.......................................................................................12
THE BERC GROUP

21 st Century Community Learning Center
Lucille Umbarger Elementary School
YEAR 4 EVALUATION REPORT
Introduction
The purpose of this report is to provide formative and summative feedback to personnel
at the Boys & Girls Club of Skagit County (BGCSC) regarding the 21 st Century
Community Learning Center (21 st CCLC) at Lucille Umbarger. This was the fourth year
of programming at this sight.
Community Context
Lucille Umbarger Elementary School served approximately 576 students in kindergarten
through eighth grades during the 2020-2021 school year. The school is located within the
Burlington-Edison School District in Burlington, Washington. Of the total school
population, over 70% identify as low income, and approximately one quarter of the
students are English Language learners. OSPI Report Card shows that for the 2018-2019
school year 1, 28% of students met ELA grade level standards, and 24% met math Grade
level standards. Community demographics are presented in Table 1.
Table 1. Lucille Umbarger Community Demographics
Federal Race/Ethnicity 2017-2018 2018-2019 2019-2020 2020-202
American Indian/Alaskan Native 1.2% 1.6% 1.3% 1.4%
1
Asian 2.1% 1.7% 1.3%
Hispanic/Latino of Any Race 55.0% 55.2% 55.6% 60.1%
Two or More Races 4.7% 4.4% 5.7% 5.4%
White 37.0% 37.1% 36.1% 31.4%
Characteristic
Low Income 69.1% 69.8% 72.8% 71.9%
English Language Learner 29.0% 28.6% 26.8% 25.3%
1 This was the most recent administration of state standardized tests. This information will be updated
when new scores are available.
THE BERC GROUP
1

Migrant 3.0% 2.5% 4.1% 4.7%
Students with Disabilities 14.0% 15.6% 15.9% 15.8%
Grant Performance Objectives
The 21st CCLC at LU continued building their program, while also adjusting to the
expectations and challenges created by the COVID pandemic. The program director
shared, “I think this year the kids just really needed to connect with one another. The kids
were fairly individual this year, with school being remote. They have missed out on the
social connections.” In addition to this targeted attention to the social-emotional needs of
students, program staff continued to focus on the grant objectives.
1. Improve student academic achievement via literacy and STEM instruction,
regular homework help and subject-specific tutoring.
2. Increase student interest in school, especially in English, Math, and Science
through project-based learning and relevant service learning
3. Improve student self-efficacy, sense of agency, social-emotional learning and
school engagement
4. Increase sense of self and personal identity through place-based development of
cultural awareness and environmental literacy
5. Increase family literacy, bilingualism, and educational aspirations.
Evaluation Design
Researchers used a multiple measures, mixed methodology approach to conduct this
evaluation. The collection of both quantitative and qualitative data adds scope and
breadth to the study in addition to providing the ability to triangulate findings (Creswell,
1994). Additionally, researchers followed a utilization-focused model of evaluation. This
model of evaluation is intended to provide useful feedback that can be implemented in
real time (Patton, 2013). Qualitative data was gathered during program observations,
interviews with staff and document review. The BGCSC provided a database of student
attendance in programming. State testing was not administered during the 2019-2020 or
2020-2021 school years due to the ongoing pandemic.
Evaluation Questions
The evaluation questions were designed to provide program leaders formative feedback
for continuous improvement, as well as summative feedback about progress toward
2
THE BERC GROUP

BGCSC 21 st Century After-School Program grant goals. The questions align with OSPI’s
local evaluation guidelines for 21 st Century Community Learning Center programs and
specific performance objectives of the BGCSC. The evaluation questions are:
Implementation Questions (Focused on the effectiveness of plans, processes, and
program strategies related to the grant vision and goals):
1. To what extent were components of the grant implemented with fidelity?
2. What challenges emerged during Year 3 implementation?
3. What promising practices emerged during Year 3 implementation?
Outcome Questions (Focused on meeting local, state, and federal performance goals and
objectives):
4. To what extent does program participation relate to improved academic and social
outcomes?
5. To what extent are program components sustainable and continuously improving?
Data Sources
To address these questions researchers gathered data from multiple sources throughout
the evaluation cycle. BGCSC provided a database of student attendance 2. Additionally,
BERC researchers conducted:
• Collection and analysis of interviews in Spring 2021
• Analysis of After-school program student attendance data
• Collection and analysis of School Age Program Quality Assessment (SA-PQA)
• Observation of the program
• Initiative documents and materials review
Implementation Questions
The following evaluation questions focus on program implementation during Year 4,
including the effectiveness of plans, processes, and program strategies relating to the
grant vision and goals. These questions were primarily assessed through staff interviews,
observations, and ongoing correspondence with program stakeholders.
2 Achievement data will be included in future reports, once available.
THE BERC GROUP
3

Q1. To what extent were components of the grant implemented with fidelity during
Year 4?
(Focus on academic support, engagement, social emotional learning, family and
community partnerships and continuous improvement.)
During Year 4, the LU 21 st CCLC staff continued to focus on programming aligned with
the values and mission of the B&G Club, while also navigating the limitations placed
upon them as a result of the COVID pandemic. Program staff demonstrated a clear
commitment to serving students by providing high-quality academic and enrichment
opportunities, despite low attendance throughout the year. The program director shared,
“I am really impressed with the quality and structure of the programming. The staff have
it together. [They have a] community wall with the stars, the interactions are really
positive, and the staff is working through and resolving issues consistently with kids.
[They] have been really successful with SEL and behavioral interventions.”
When asked to discuss their academic programming, one staff member outlined their
typical process. She shared,
We ask kids to do homework first, then reading, then power pages. I have a folder
for each with grade appropriate content that targets the skills they need to work
on. Sometimes they have self-directed projects that they need to finish from class.
We have a power point system- there is a grid on the board of 250 power pointseach
assignment has a point value- and they can earn bonus points. They get to
initial one of the squares, and they can work together to earn a party. We used to
do it individually, but it has seemed to work better for this group of kids to work
together.
Staff members noted that they continued to reach out to teachers for student information
to better address individual needs. One staff member shared, “If a kid is falling behind, a
teacher might come to us and ask us if we can help them. [Our site coordinator] has been
good about talking with the staff, and has talked to the principal and front office. We are
talking about being more intentional, and thinking ahead for next year.”
In addition to academic support, the 21st CCLC staff provided enrichment opportunities
focused on social emotional learning and issues of diversity, equity, and inclusion.
Students participated in a 6-week curriculum from Youth for Unity, which taught students
about unconscious bias and white privilege. Staff also worked to incorporate SEL into
everything they did in an effort to build one another up and promote positive self-
4
THE BERC GROUP

awareness. Staff implemented The Community Corner, where they would post their
program values and give one another ‘shout outs’. One staff member shared, “[We are]
trying to build community. We have a lot of kids working on self- regulation and [setting]
goals for themselves. A lot of SEL just happens naturally, but [we] have pulled a little
from Sanford and Harmony.” The site coordinator noted, “We have a lot of success when
the staff finds things that are specific to their students. We are not just relying on those
curricula. We have a lot of resources, and are modifying them to meet the needs of the
kids in the moment.”
Q2. What challenges emerged during Year 4 implementation?
Program leadership and staff shared that most of the challenges during Year 4 were
directly related to the pandemic. COVID scheduling was difficult, as things would open,
then close, making it so the “window of opportunity did not always coincide with the
state and district regulations.” One staff member discussed their i-clubhouse program that
was implemented during the initial COVID school closures in 2019-2020, but noted that
they tried carrying the program into the fall, and it didn’t get traction once students were
learning from home. She felt “it was too much computer time, and families weren’t really
into it.” As a result of these challenges, student attendance was low. One staff member
noted, “One of the attendance issues is that we still had to operate after school, [and] it
has been much more difficult for families to get here after school if the kids weren’t in
school full day.”
In addition to Covid and attendance, staffing remained a persistent challenge for LU. The
site coordinator shared, “Here all of our part time staff are school district employees.
There are opportunities for district staff to grow through the school district, so we lose
staff sometimes to other opportunities.”
Q3: What promising practices emerged during Year 4 implementation?
The 21 st CCLC program at LU has had challenges over the past few years related to
program and staff stability. Despite these difficulties, program leaders and staff felt that
they had developed a strong program for the students who were able to attend, and
dedicated time and energy to providing the social, emotional and academic support their
students needed during the pandemic. LU 21 st CCLC staff were organized, dedicated to
building relationships, and successful at creating routines and rituals to help create an
engaging and enriching program.
.
THE BERC GROUP
5

Impact Questions
Q4: 4. To what extent does program participation relate to improved academic and
social outcomes?
During interviews staff were asked to share their perceptions of student outcomes during
Year 4. Overall, staff noted that students seemed to benefit most from opportunities to
engage with their peers again after so much time isolated from friends and non-family
adults. Although it was difficult for staff to say with certainty whether students made
educational gains during Year 4, they did feel confident that students had developed more
adaptive social-emotional skills, and were more able to build relationships and ask for
what they needed because of the individualized attention and support they received
throughout the year.
In addition to anecdotal evidence of student academic and social outcomes, researchers
analyzed attendance data for students participating in the 21 st CCLC. As a result of the
school closures, no state assessment scores or discipline data were available for analysis.
Attendance. One outcome measure for the 21 st Century program is student attendance.
Program leaders collected and shared program attendance for all participating students.
Students were counted if they attended at least one day of program. Students that were in
attendance for more than 30 days were labeled “21 st Century” students. During Year 4,
LU had 23 students register, and 10 students attend at least 30 days (Table 2). This was a
sharp drop compared to the truncated 2019-2020 school year and was likely attributed to
distance learning and pandemic concerns.
Table 2. Lucille Umbarger Elementary Attendance
2019-2020 2020-2021
Number of Participating Students 60 23
Number of 21st Century Students 23 10
Percentage of 21st Century Students 38% 44%
When disaggregated by month, the total attendance was steady throughout the year,
between 8 and 11 students for most months (Figure 1).
6
THE BERC GROUP

Figure 1
Q5. To what extent are program components sustainable and continuously
improving?
Evaluators will continue to gather and track outcomes for grant objectives over time.
Although this report summarizes program efforts in Year 4, this site has had changes and
challenges that have impacted the timeline of progress, making longitudinal data more
difficult to track. Despite these challenges, the program staff and operating organization
are committed to serving the students, and have made significant changes intended to
strengthen the program and build a stronger connection to the school district and
community. In addition to the efforts to build a foundation for this 21 st CCLC, staff and
leadership also engaged in professional development and continuous improvement
efforts, including the Youth Program Quality Assessment.
Youth Program Quality Assessment
To fulfill grant requirements, researchers used the YPQA (Youth Program Quality
Assessment) tool to evaluate the quality of youth experiences in the program. This tool
assesses four main domains of programming, including safe environment, supportive
environment, interaction, and engagement. Youth-centered policies and practices, high
THE BERC GROUP
7

expectations for youth and staff, and access were other areas incorporated into the tool.
The YPQA was developed by the Weikart Center, and according to their website (2014),
“each domain contains items that focus on specific elements of best practice.” The
findings provide opportunities for staff members to reflect on what they are doing well, to
determine ways they can improve the program, and “to build professional competencies.”
Although the YPQA was not required during the 2020-2021 school year, results from
Year 3 are included to support the LU 21 st CCLC staff as they develop goals for their
future.
The overall Domain scores for Interaction and Engagement, which measure opportunities
for students to collaborate, lead, plan, and reflect, were low for the self and external
assessments, while Safe Environment scored high during both assessments (Figure 3).
4.6
4.6
LU YPQA Domain Scores
3.7
3.5
2.7
2.5
2
2
Safe Environment Supportive Environment Interaction Engagement
Figure 3
Self-Assessment
External Assessment
When disaggregated into individual scales, opportunities for leadership within the
Interaction domain scored at 1.67 out of 5, while collaboration, belonging and adult
partnerships scored higher (Figure 4). Within the Engagement domain, planning and
reflection were the lowest scoring scales (Figure 5).
8
THE BERC GROUP

LU YPQA Interaction Scale Scores
3
3 3
3
2.3
1.7 1.7
3
Belonging Collaboration Leadership Adult Partners
Figure 4
Self-Assessment
External Assessment
LU YPQA Engagement Scale Scores
3
3.5
1
1
2
1.5
Figure 5
Planning Choice Reflection
Self-Assessment
External Assessment
Opportunities for growth identified by self and external assessments include developing
student leadership capacity, building cooperation, and incorporating reflection into
learning activities. The Center for Youth Program Quality provided the following
recommendations to support further program growth:
o
Cooperative Learning: Providing young people an opportunity to participate in
and lead small groups has a positive impact on classroom climate, self-esteem
among students, internal locus of control, and time on task. Students in
THE BERC GROUP
9

cooperative teams are more active, self-directing, and expressive, all of which
may be associated with achievement gains.
o
Youth Voice: Providing young people with chances to make decisions about their
activities and how they carry them out can improve motivation and buy-in, and
more importantly, offering choices in a youth program space gives youth a chance
to practice for the bigger choices they'll make outside of the program.
o
Planning and Reflection: The skills of making plans for the future and learning
from the past can help youth succeed in school and in life. These skills are tied
into what brain scientists call executive functions, and play an important role in
directing attention to tasks and decision making that connects with consequences.
Conclusions and Recommendations
Throughout Year 4, the LU 21 st CCLC made progress toward many of the grant goals and
objectives. To continue to make improvements, we suggest the following
recommendations:
Collect and Use Data
Program leaders at each site should continue to document and collect data required for
the program and the evaluation. Additional training on the YQPA, as well as regular time
to perform and reflect on the tool should be allocated. This data provides critical, ongoing
information on student engagement and the effectiveness of enrichment activities, which
can be used to inform and improve the LU 21CCLC program. Additionally, the use of
real time student data will help program staff to help students to track assignments and to
be academically prepared. The sharing of data with the school, family and outer
community members may help to create a strong system of support, awareness, and
recognition for the program.
Strengthen Community and Family Partnerships
The LU 21 st CCLC program would benefit from increased community involvement and
support. By strengthening community partnerships, students engage in new experiences
and develop a broader world view. New program leaders may want to collaborate with
previous grant staff to gain insight about partnerships that are already established while
actively working to reach out to form new community collaborations.
10
THE BERC GROUP

THE BERC GROUP
11

Prepare for program growth and think about long term sustainable plans
This year, the program grew into a five day-a-week program serving expanded grades.
This growth created some challenges associated with funding, the availability of
resources, and staffing. Program leaders will want to plan for future program attendance
numbers and ensure the budget and staff ratio can appropriately accommodate the
growing needs of the program. As the program continues, staff members will also want to
think about ways they can sustain efforts beyond the availability of grant money.
Increase student choice, voice, cooperative learning, and reflection in projects
Recommendations from the YPQA suggest that site staff should increase opportunities
for student planning and incorporate student reflection into regular programming.
According to the YPQA report, “The skills of making plans for the future and learning
from the past can help youth succeed in school and life.” Ways to increase student
reflection include providing a dedicated reflection time at the start or end of the day,
asking three reflection questions at the end of each session, and by incorporating
journaling, partner talk, and exit slips. Staff and students alike may benefit from learning
strategies about how to ask reflective questions and about how to answer such questions.
Students may feel more invested in the program when they feel listened to and are an
active part of program planning. According to the YPQA, providing youth the chance to
make decisions about their activities and how they carry them out “can improve
motivation and buy in. Offering choices in the solace of a youth program space gives
youth a chance to practice for the bigger choices they’ll make outside of the program.”
Students may feel more invested in the program if they are regularly asked to share their
insights about projects and are encouraged to plan future activities. Lastly, it may
behoove staff to work on ways to incorporate opportunities for students to lead and
participate in small group learning. “Cooperative learning promotes student voice, allows
student to take on leadership roles, and “has a positive impact on classroom climate, selfesteem
among students, internal locus of control, and time on task” (YPQA, 2014).
References
David P.Weikart Center for Youth Program Quality. Youth Program Quality Assessment
and School-Age Program Quality Assessment. Obtained 8/27/2014 from http://
www.cypq.org/assessment
12
THE BERC GROUP

THE BERC GROUP
13

The BERC Group, Inc.
22232 - 17 th Ave. SE Suite 305
Bothell, WA 98021
Phone: 425.486.3100
Web: www.bercgroup.com
14
THE BERC GROUP