Concrete_21st Century Grant_Year 3_Evaluation2021

EVALUATION REPORT 2021
Concrete School District
21 st Century Community Learning
Centers: Cohort 15
YEAR 3 EVALUATION REPORT
STACY MEHLBERG, Ph.D.
RONAK PATEL, Ph.D.

Duane Baker is the founder and president of Baker
Evaluation, Research, and Consulting, Inc (The BERC
Group). Dr. Baker has a broad spectrum of public school
educational and program experience, including serving as a
high school classroom teacher, high school assistant principal,
middle school principal, executive director for curriculum and
instruction, and assistant superintendent. In addition, he has
served as an adjunct instructor in the School of Education at
Seattle Pacific University since 1996, where his emphasis has
been Educational Measurement and Evaluation and
Classroom Assessment.
Dr. Baker also serves as the Director of Research for the
Washington School Research Center at Seattle Pacific
University. He also serves as an evaluator for several
organizations including the Bill & Melinda Gates Foundation,
Washington Education Foundation, Washington State Office
of Superintendent of Public Instruction, and others.
Members of The BERC Group have K–20, experiences as
teachers, counselors, psychologists, building administrators,
district administrators, and college professors. The team is
currently working on research and evaluation projects at the
national, state, regional, district, school, classroom, and
student levels in over 1000 schools in Washington State and
nationally.
COPYRIGHT © 2020 BY THE BERC GROUP INC. ALL RIGHTS RESERVED
ADDITIONAL COPIES OF THIS REPORT MAY BE OBTAINED THROUGH THE BERC GROUP (www.bercgroup.com).
THE BERC GROUP

Table of Contents
Concrete School District 21st Century Community Learning
Centers...............................................................................................1
Introduction .......................................................................................1
Grant Performance Objectives .......................................................................2
Evaluation Design .............................................................................3
Evaluation Questions ......................................................................................3
Data Sources ....................................................................................................4
Overall Findings ................................................................................4
Youth Program Quality Assessment ...............................................9
Conclusions.....................................................................................13
Recommendations ..........................................................................14
References.......................................................................................17
THE BERC GROUP

Concrete School District 21 st Century
Community Learning Centers
Year 3 Evaluation Report
Introduction
Beginning in 2018, the Boys & Girls Clubs of Skagit County (BGCSC) partnered with
the Concrete School District to expand 21 st Century Community Learning Center (21 st
CCLC) programming to youth and teens in eastern Skagit County. BGCSC has been
active in the local community for several years, and currently supports seven 21 st CCLCs
throughout their service area. Although Concrete School District did not meet the
requirement for racial diversity, with less than 15% of their population identifying as nonwhite,
program leaders worked for 18 months to secure funding by using data and
research to articulate the needs of this small school district serving approximately 500
students.
The mission of the BGCSC is to “enable all young people, especially those who need us
most, to reach their full potential as productive, caring, responsible citizens” (BGCSC
Annual Report, 2018). The organization prioritizes three areas: Academic Success,
Healthy Lifestyles, and Good Character and Citizenship. The main goals of the program
include providing services and promoting academic growth for low income and high need
students within the Concrete school district. The grant proposal states a range of expected
outcomes, including:
• Improved academic performance
• Enriched learning experiences
• Increased family engagement
The 21 st CCLC in the Concrete School District has operated at two sites: Concrete
Elementary School and Concrete Middle School. There is a program director to manage
both sites, with between seven and nine support staff for the two programs. Both schools
are school-wide Title I schools with somewhat similar demographics (Table 1). Concrete
high school had an enrollment of 202 students during the 2020-2021 school year, while
the elementary school had 292 students enrolled.
THE BERC GROUP
1

Table 1
Demographics of 21 st Century Community Learning Center Schools.
Ethnicity
American Indian/
Alaskan Native
Elementary
2018-19 2019-20 2020-2
1
Secondary
2018-19 2019-2
0
2020-21
3.1% 3.1% 4.1% 0.9% 0.9% 1.5%
Asian 0.4% 0.4% 0.7% 1.4% 0.9% 1.0%
Black 0.4% 0.4% 0.0% 0.0% 0.0% 0.0%
Hispanic 11.0% 11.0% 11.3% 11.0% 11.0% 9.4%
White 81.0% 81.0% 80.5% 87.0% 85.6% 87.1%
Two or More 3.9% 3.9% 3.4% 0.0% 1.4% 1.0%
Free or Reduced Lunch 93.0% 92.7% 84.9% 87.0% 86.6% 79.2%
Grant Performance Objectives
The specific 21 st CCLC grant goals identified in the original proposal are listed below.
• Goal 1: Improve academic school related outcomes
• Goal 2: Improve positive student behavior, engagement and healthy choices
• Goal 3: Provide family engagement, development, and education strategies
• Goal 4: Utilize a continuous quality improvement program environment
The purpose of this report is to provide formative and summative feedback to personnel
at the BGCSC and OSPI regarding the third year of the 21 st Century Community
Learning Centers (21 st CCLCs). Although summative in nature regarding the impact of
the program on the staff, parents, and students involved, the report is also designed to
provide formative feedback to program leaders. The report begins by providing general
information about the goals and objectives of the program. The introduction is followed
by a description of evaluation activities, evaluation findings, and recommendations.
Logic Model
2
THE BERC GROUP

Beginning with the 2019-2020 school year, each program site worked to create a Logic
Model to support their vision and help drive decision making. Program leaders and site
staff worked with the external evaluator to discuss the current and future needs of
students and community members. An example of Concrete’s 2020-2021 logic model is
included in Appendix A. This process will be conducted with evaluation teams for all
future years of the grant.
Evaluation Design
Researchers used a multiple measure, mixed methodology approach to conduct this
evaluation. The collection of both quantitative and qualitative data adds scope and
breadth to the study in addition to providing the ability to triangulate findings (Creswell,
1994). During the 2020-2021 school year, school closures and COVID restrictions
impacted the ability to collect quantitative data from program sites, as state assessments
were cancelled. As a result, the Year 3 report will look different than prior and future
reports. Program leaders were asked to provide their perceptions of the impact of COVID
on programming and student outcomes. Those perspectives are included in the evaluation
narrative.
Despite school closures, researchers followed a utilization-focused model of evaluation.
This model of evaluation is intended to provide useful feedback that can be implemented
in real time (Patton, 2013). As such, The BERC Group worked closely with program
leaders to develop evaluation questions and design. Qualitative data was gathered during
program observations, interviews with staff, and document review. The Boys and Girls
Club provided a database of student attendance for analysis by researchers.
Evaluation Questions
To address program goals and grant objectives, researchers collaborated with the program
leader to assure alignment of evaluation questions with the goals and requirements of the
grant. The evaluation questions were designed to provide program leaders formative
feedback for continuous improvement, as well as summative feedback about progress
toward grant goals. The questions align with OSPI’s local evaluation guidelines for 21 st
CCLC programs. The evaluation questions are:
1. What strategies and activities support project goals?
THE BERC GROUP
3

2. What contextual factors influence program implementation?
3. To what extent does the program reach the target population?
4. To what extent do students persist in the program?
5. Outcome Questions:
a. To what extent does the program contribute to improved math and reading
scores?
b. To what extent is attendance at the program correlated with academic
achievement?
c. To what extent does the program enrich students’ learning experiences?
d. To what extent do families engage with the program?
As a result of COVID school closures, question 5 parts a and b were not addressed during
this Year 3 evaluation. Researchers did ask program participants to share their
perspectives of student outcomes during the 2020-2021 school year. These perspectives
are discussed throughout the report.
Data Sources
To address these questions researchers gathered data from multiple sources throughout
the evaluation cycle. BGCSC provided a database of student attendance. Additionally,
BERC researchers conducted
• Program interviews with staff and leadership
• Program observation
• A review of initiative documents and materials
Overall Findings
EQ1. What strategies and activities support project goals?
During Year 3, the Concrete 21 st CCLC met on-site, utilizing a 1:10 staff to student ratio
to cohort students. Compared to prior years, attendance was low, with approximately
8-12 students participating regularly between the elementary and secondary cohorts. Both
groups met in the BGCSC trailer, utilizing separate rooms. The program director shared
that they were trying to maintain consistency with prior years, but the poor attendance
4
THE BERC GROUP

made that challenging. One staff member commented, “We were getting 1 kid a day on
the elementary side, [and] a few on the high school side, so our big push has been to try
to bring more kids in, in a safe way. A lot of this has come down to reaching out to
families…Bringing those kids in really breathed new life into the club. One kid is fine,
but it is not really a club environment.” He continued to note that as a few more students
engaged, they were able to offer more enrichment and academic programming.
Program leaders and staff expressed pride in their programming this year, noting that they
focused on Social Emotional Learning (SEL) to meet their goals around student behavior,
family connection, and academic support. Students, they proposed, were struggling to
participate in school, and needed more support and guidance on how to re-acclimate to
social and academic pressures and expectations. Once attendance increased slightly,
program staff offered a range of activities, keeping in mind the restrictions in place for
social distancing. One staff member noted, “I believe [SEL] is a strong foundation for
academics, and has been more of a focus this year because of COVID. [Our] kids coming
to club after hours and hours on the computer was rough. Our ability to help those kids
build coping mechanisms was huge. COVID gave us an excuse to really focus on it, and I
hope we can keep it going, since we have had great results.”
Program offerings included Power Hour, which did not look “traditional,” but staff felt
was critical to building rapport and relationships with students. Staff also noted that
students were really struggling with their academics, and seemed to benefit from the
individualized attention that 21 st CCLC staff were able to provide due to the low
attendance numbers. In addition to Power Hour, program staff partnered with the Farm to
School program, which provided experiences for kids to engage in hands on learning.
Students got to spend time outside in their garden, learning to farm, and incorporating
plant learning and nature into their activities and experiments. One staff member also
discussed their reading buddy program, which matched older students with an elementary
partner to improve reading skills. He shared,
We also have been trying to balance the needs of our different ages. Our two
Kindergarten students were struggling with reading, so we started to have some of
the older kids act as buddies for the littles, helping them to read. This was part of
the SEL work, but it was also academic support and leadership opportunities for
the older students. The older students loved it. Giving them roles made them feel
important, they felt needed, and they were honestly doing important things. They
enjoyed getting to bond, and feel important.
THE BERC GROUP
5

In response to COVID and overall student well-being, the 21 st CCLC staff introduced a
Healthy Habits program with their students, as hygiene and personal space became such
an impactful part of the club experience. Staff conducted a series of lessons focused on
hygiene using the Healthy Habits curriculum. One staff commented, “We gave them the
space to know that beyond all of the COVID restrictions, [there are hygiene aspects] we
do every day. The kids responded pretty positively.” Staff also spoke highly of the
organized and thorough COVID protocols that were in place, making it possible for
students to participate safely throughout the year without being anxious about getting
sick.
EQ2. What contextual factors influence program implementation?
During Year 3, factors that impacted the ability for program to operate at capacity
included issues around staffing, transportation, technology, and COVID restrictions.
There were several staffing challenges during the 2020-2021 school year that impacted
programming. Roles within the BGCSC changed, and leadership of the 21 st CCLC shifted
accordingly. Additionally, it was a challenge to find staff interested in working in the
program, due to location and salary. The program director noted, “[Our] location makes
staffing an issue. Also, and just a speculation, but with unemployment being so generous,
we are all having trouble getting staff, since we can hire just above minimum wage.”
Program staff also commented on challenges around staffing, however, they felt that they
had been able to experience success as a team. One staff member shared,
We work well together. We have been able to build up our club culture, and create
an environment we want to work in. In some ways starting from the ground up
has been really positive. We have a really great starting point with great attitudes.
[Staff] are rolling with the punches…we have been flexible and understanding of
the complicated time we are in. Even though we are a small team, we run
ourselves effectively and confidently.
In addition to challenges with hiring and retaining staff, the lack of transportation
available to students was a significant barrier throughout the year. Students who were not
in school did not have a way to participate. The program director commented, “With
transportation, we would have had more kids and more robust programming.”
Technology use was another challenge during Year 3. Program staff expressed their
concerns about the impact of so many hours in front of computer screens during the
pandemic, noting effects on students’ social, emotional, and academic well-being. One
staff member shared, “We need to get kids unplugged, [and] get them outside to get some
6
THE BERC GROUP

exercise. A lot of these kids are sitting around for hours and hours every day. We are
really focusing on getting outside as much as we can, and there is some push back since
some of them like to be on their computers. [But] when the kids are outside, they are
being creative.”
Finally, the pandemic created new challenges to implementing 21 st CCLC programming.
Staff recognized the time and resources dedicated to complying with COVID policies and
procedures. They also acknowledged that it required creative thinking to develop and
implement COVID-safe programming. Staff shared several examples of games they
played during the year, including pool noodle tag, Ship to Shore, and Red-Light Green
Light. The program director shared, “some games were familiar, but they were all created
fresh since the old challenges weren’t COVID compliant.”
EQ3. To what extent does the program reach the target population?
The BGCSC 21 st CCLCs were intentionally placed in this community to provide social,
emotional, and academic enrichment to as many students as possible throughout the
service area. Since approximately 80% of the students in Concrete School District receive
free/ reduced lunch benefits, program leaders knew they would be serving their target
population. During Year 3, program staff were aware of the unique challenges impacting
their students. Although typically a robust program, it was difficult to maintain a steady
and strong attendance this year, due to COVID, transportation issues, and capacity of
space and staffing. The program continued to make an effort to reach their target
population, however, strengthening their partnership with the school district, and
maintaining their social capital in the community.
EQ4. To what extent do students persist in the program?
Researchers analyzed attendance data to better understand student engagement and
persistence in the 21 st CCLCs during Year 3. Figure 1 illustrates the average days of
attendance throughout the school year for students who were present for at least one day
of the program. At the elementary site, students attended an average of 4 days while at the
secondary site students attended an average of 2 days. It should be noted that the low
attendance is a result of the ongoing pandemic, greatly impacting the overall average
days attended. When disaggregated by month, July and August had higher rates of
attendance at both the elementary and secondary sites, with a harsh dip in the winter and
a slight increase in the spring.
THE BERC GROUP
7

Average Days of A-endence
Elementary
Secondary
53
49
32
30
4
2
Figure 1
2018-19 2019-20 2020-21
Students attending at least 30 days of the program throughout the year were given awards
and identified as ‘21 st Century Students.’ The number of 21 st Century students at each site
are displayed in Figure 2.
70
Number of 21st Century Students
59
Elementary
76
Secondary
17
7
4
Figure 2
2018-19 2019-20 2020-21
Figure 3 illustrates the percent of 21 st Century students among the total number of
program participants. Again, students were considered in the total if they attended at least
8
THE BERC GROUP

1 day of the program in the Spring or Fall session. Of the total counted, 32% of
elementary students earned 21 st Century status and 40% of secondary students earned 21 st
Century status.
Percentage of 21st Century Students
Elementary
Secondary
57%
61%
33%
31% 32%
40%
Figure 3
EQ5A. To what extent does the program contribute to improved math and reading
scores? Due to the suspension of state standardized tests and distance learning for most
of the 2020-2021 school year, researchers were unable to collect state assessment scores.
This research question will be revisited once assessments resume, hopefully in 2022.
Youth Program Quality Assessment
2018-19 2019-20 2020-21
To fulfill grant requirements, researchers used the SA-PQA (School-Aged Program
Quality Assessment) and YPQA (Youth Program Quality Assessment) tools to evaluate
the quality of youth experiences in the program. These assessments measure four main
domains of programming, including safe environment, supportive environment,
interaction, and engagement. Youth-centered policies and practices, high expectations for
youth and staff, and access were other areas incorporated in the tool. The YPQA was
developed by the Weikart Center, and according to their website (2014), “each domain
contains items that focus on specific elements of best practice.” The findings provide
opportunities for staff members to reflect on what they are doing well, to determine ways
they can improve the program, and “to build professional competencies.”
Figure 4 and displays the overall domain scores for the self-assessments conducted at
Concrete High School during Year 1 (2018-2019) and Year 2 (2019-2020). Programs did
not participate in the YPQ assessments during Year 3, but will resume in Year 4.
Self-assessments are conducted internally by trained program staff. The scores reflect
THE BERC GROUP
9

observations from the Power Hour, Positive Action, and Smart Girls learning experiences
during Year 1, and Positive Action and art experiences during Year 2. Across all domains,
self-assessment scores in Year 2 decreased.
Concrete High School YPQ Self Assessment Over Time
4.6
4.53
4.23
3.9
3.08
2.88
3.67
2.67
Figure 5
Safe Environment Supportive Environment Interaction Engagement
2019
2020
Figure 5 shows two years of scores for external assessments, conducted by BERC
evaluators at Concrete High School in December 2019. External assessor scores for
Supportive Environment and Interaction were lower in 2020, while scores in Safe
Environment remained consistent, and scores in Engagement increased. Scores in
engagement have been the lowest across years and between external and selfassessments.
Concrete High School YPQ External Assessment Over
Time
4.9
4.83
4.5
3.74
4.02
3.15
2.17
2.33
Safe Environment Supportive Environment Interaction Engagement
Figure 6
2019
2020
10
THE BERC GROUP

In a similar trend to Year 1, several individual items scored low (below 3.0 out of 5.0) on
both assessments, including opportunities for students to collaborate, develop their
leadership capacity, make choices about their learning, and reflect on their experiences.
Recommendations from The Center for Youth Program Quality included a focus on the
domains of Interaction and Engagement:
o
Cooperative Learning: Providing young people an opportunity to
participate in and lead small groups has a positive impact on classroom
climate, self-esteem among students, internal locus of control, and time on
task. Students in cooperative teams are more active, self-directing, and
expressive, all of which may be associated with achievement gains.
o
Youth voice: Providing young people with chances to make decisions
about their activities and how they carry them out can improve motivation
and buy-in, and more importantly, offering choices in a youth program
space gives youth a chance to practice for the bigger choices they'll make
outside of the program.
o
Ask, listen, encourage: Positive relationships can open the gateway to
learning. Studies find that relationships with "warmth, connectedness,
good communication and support" aid in positive youth development and
are connected to academic success (National Research Council and the
Institute of Medicine, 2002). Asking effective questions, listening to
youth, and encouraging youth through positive and specific feedback can
help to build positive relationships and may influence the intrinsic
motivation of youth.
Figure 6 displays the overall domain scores for the self-assessments conducted at
Concrete Elementary School in December 2019. Scores reflect observations from the
Power Hour, Triple Play, Lego Robotics, Project Read, and Art learning experiences.
Scores in 2019 were lower in all four domains, with the greatest difference in
Engagement.
THE BERC GROUP
11

Concrete Elementary School SA-PQ Self Assessment Over
Time
4.07
3.84
3.79
3.56 3.45
3.46
3
2.08
Figure 7
Safe Environment Supportive Environment Interaction Engagement
2019
2020
External assessor scores for the December 2019 program visits were higher than during
Year 1 in all domains except Safe Environment, which decreased only slightly between
years (Figure 7). Additionally, external assessor scores were higher than self-assessment
scores for each domain.
Concrete Elementary School SA-PQ External Assessment
Over Time
4.9
4.79
4.25
4.41
3.37
3.28
2.42
2.54
Safe Environment Supportive Environment Interaction Engagement
Figure 8
2019
2020
Opportunities for growth identified by both self and external assessments include creating
more opportunities for students to have voice and choice in their learning, incorporating
12
THE BERC GROUP

reflection, and helping students to develop skills to plan activities around their learning.
The Center for Youth Program Quality provided the following recommendations to
support further program growth:
o
Youth voice: Providing young people with chances to make decisions
about their activities and how they carry them out can improve motivation
and buy-in, and more importantly, offering choices in a youth program
space gives youth a chance to practice for the bigger choices they'll make
outside of the program.
o
The skills of making plans for the future and learning from the past can
help youth succeed in school and in life. These skills are tied into what
brain scientists call executive functions, and play an important role in
directing attention to tasks and decision making that connects with
consequences.
Conclusions
Over the past three years, the Concrete school district and Boys & Girls Club have
developed a sustained and committed partnership to serve students through their 21 st
CCLCs. Students have benefit from programming that supports their social, emotional,
and academic needs, while also attempting to engage them in critical and creative
thinking, community action, and persistence through difficulties.
Concrete faced challenges this year, including staff retention, COVID restrictions, and
limits to transportation resulting in poor attendance. Despite these challenges, staff shared
that they felt proud of their work, and were able to develop meaningful relationships with
students. Program leaders also shared that they were excited for their summer program,
which would look different due to COVID, but provide them with unique opportunities to
support students more holistically. One program staff member shared, “One positive for
us is that since Concrete is doing an in-depth summer school program, we can work
closely with them to help kids catch up a little, and get ready for next year. We will get to
work with the teachers, and see where the kids are at, and where they are struggling the
most.”
THE BERC GROUP
13

Recommendations
Year 3 results provided evidence to support the value of the program in helping students
to make build relationships with adults, and develop important social emotional skills
needed to be successful in school. To continue to make improvements, we suggest the
following recommendations, which are a continuation of a more typical year of
programming:
Continue to build capacity for community and family partnerships.
A strong focus of the program has been cultivating positive partnerships with community
and family members. These partnerships should continue to be sought out and built upon
in the future, as they result in more positive experiences for students. One
recommendation is to expand the scope of partnerships to increasing volunteer
opportunities and service-learning projects, giving students a chance to explore their
interests in the community. Allowing student voice and choice is aligned with the
program mission and vision and will likely increase student engagement while providing
valuable, action-oriented experiences for students.
Increase student reflection in projects.
Results from the YPQA suggest that program leaders should increase opportunities for
student reflection into regular programming. Recommendations from the YPQA notes
“The skills of making plans for the future and learning from the past can help youth
succeed in school and life. These skills are tied into what brain scientists call executive
functions and play an important role in directing attention to task and decision making
that connects with consequences” (YPQA, 2017). Providing dedicated time, at the start or
end of an activity, gives students the opportunity to regularly reflect on their learning
throughout the program. These reflections can be verbal, written, or even illustrative, and
should include sharing their thoughts with peers. For example, students can be given exit
slips at the end of each program session that ask reflection questions such as “Today I
Learned….” Program leaders can also use this information to improve the student
experience. In light of current pandemic-based restrictions, these exit slips, or any
reflections, can be given online, through Google or Microsoft Forms.
Collect and Use Data
14
THE BERC GROUP

Program leaders at each site should continue to document and collect data required for
the program and the evaluation. Additional training on the YQPA, as well as regular time
to perform and reflect on the tool should be allocated. This data provides critical, ongoing
information on student engagement and the effectiveness of enrichment activities, which
can be used to inform and improve the 21 st CCLCs. Additionally, the use of real time
student data will help program staff to better meet the needs of students. The sharing of
data with the school, family and outer community members may help to create a strong
system of support, awareness, and recognition for the program.
Continued Focus on Program Activities and Expectations that Align with the Vision
It is important that program staff members continue to plan activities early in the year, in
alignment with the vision and goals of the program. Program leaders should continue to
find innovative ways to communicate goals and activities to general education teachers to
ensure all stakeholders are operating under the same beliefs. One suggestion is to share
the program’s logic model with all stakeholders, so they can see and have discussions
about the best ways to collaborate for the benefit of the students.
THE BERC GROUP
15

Appendix A
2019-2020 Logic Model
16
THE BERC GROUP

References
David P.Weikart Center for Youth Program Quality. Youth Program Quality Assessment
and School-Age Program Quality Assessment. Obtained 8/27/2014 from http://
www.cypq.org/assessment
THE BERC GROUP
17

The BERC Group, Inc.
P.O. Box 3552
Redmond, WA 98052
Phone: 425.486.3100
Web: www.bercgroup.com
18
THE BERC GROUP