11.08.2022 Views

Concrete_21st Century Grant_Year 3_Evaluation2021

Create successful ePaper yourself

Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.

EVALUATION REPORT 2021

Concrete School District

21 st Century Community Learning

Centers: Cohort 15

YEAR 3 EVALUATION REPORT

STACY MEHLBERG, Ph.D.

RONAK PATEL, Ph.D.


Duane Baker is the founder and president of Baker

Evaluation, Research, and Consulting, Inc (The BERC

Group). Dr. Baker has a broad spectrum of public school

educational and program experience, including serving as a

high school classroom teacher, high school assistant principal,

middle school principal, executive director for curriculum and

instruction, and assistant superintendent. In addition, he has

served as an adjunct instructor in the School of Education at

Seattle Pacific University since 1996, where his emphasis has

been Educational Measurement and Evaluation and

Classroom Assessment.

Dr. Baker also serves as the Director of Research for the

Washington School Research Center at Seattle Pacific

University. He also serves as an evaluator for several

organizations including the Bill & Melinda Gates Foundation,

Washington Education Foundation, Washington State Office

of Superintendent of Public Instruction, and others.

Members of The BERC Group have K–20, experiences as

teachers, counselors, psychologists, building administrators,

district administrators, and college professors. The team is

currently working on research and evaluation projects at the

national, state, regional, district, school, classroom, and

student levels in over 1000 schools in Washington State and

nationally.

COPYRIGHT © 2020 BY THE BERC GROUP INC. ALL RIGHTS RESERVED

ADDITIONAL COPIES OF THIS REPORT MAY BE OBTAINED THROUGH THE BERC GROUP (www.bercgroup.com).

THE BERC GROUP


Table of Contents

Concrete School District 21st Century Community Learning

Centers...............................................................................................1

Introduction .......................................................................................1

Grant Performance Objectives .......................................................................2

Evaluation Design .............................................................................3

Evaluation Questions ......................................................................................3

Data Sources ....................................................................................................4

Overall Findings ................................................................................4

Youth Program Quality Assessment ...............................................9

Conclusions.....................................................................................13

Recommendations ..........................................................................14

References.......................................................................................17

THE BERC GROUP


Concrete School District 21 st Century

Community Learning Centers

Year 3 Evaluation Report

Introduction

Beginning in 2018, the Boys & Girls Clubs of Skagit County (BGCSC) partnered with

the Concrete School District to expand 21 st Century Community Learning Center (21 st

CCLC) programming to youth and teens in eastern Skagit County. BGCSC has been

active in the local community for several years, and currently supports seven 21 st CCLCs

throughout their service area. Although Concrete School District did not meet the

requirement for racial diversity, with less than 15% of their population identifying as nonwhite,

program leaders worked for 18 months to secure funding by using data and

research to articulate the needs of this small school district serving approximately 500

students.

The mission of the BGCSC is to “enable all young people, especially those who need us

most, to reach their full potential as productive, caring, responsible citizens” (BGCSC

Annual Report, 2018). The organization prioritizes three areas: Academic Success,

Healthy Lifestyles, and Good Character and Citizenship. The main goals of the program

include providing services and promoting academic growth for low income and high need

students within the Concrete school district. The grant proposal states a range of expected

outcomes, including:

• Improved academic performance

• Enriched learning experiences

• Increased family engagement

The 21 st CCLC in the Concrete School District has operated at two sites: Concrete

Elementary School and Concrete Middle School. There is a program director to manage

both sites, with between seven and nine support staff for the two programs. Both schools

are school-wide Title I schools with somewhat similar demographics (Table 1). Concrete

high school had an enrollment of 202 students during the 2020-2021 school year, while

the elementary school had 292 students enrolled.

THE BERC GROUP

1


Table 1

Demographics of 21 st Century Community Learning Center Schools.

Ethnicity

American Indian/

Alaskan Native

Elementary

2018-19 2019-20 2020-2

1

Secondary

2018-19 2019-2

0

2020-21

3.1% 3.1% 4.1% 0.9% 0.9% 1.5%

Asian 0.4% 0.4% 0.7% 1.4% 0.9% 1.0%

Black 0.4% 0.4% 0.0% 0.0% 0.0% 0.0%

Hispanic 11.0% 11.0% 11.3% 11.0% 11.0% 9.4%

White 81.0% 81.0% 80.5% 87.0% 85.6% 87.1%

Two or More 3.9% 3.9% 3.4% 0.0% 1.4% 1.0%

Free or Reduced Lunch 93.0% 92.7% 84.9% 87.0% 86.6% 79.2%

Grant Performance Objectives

The specific 21 st CCLC grant goals identified in the original proposal are listed below.

• Goal 1: Improve academic school related outcomes

• Goal 2: Improve positive student behavior, engagement and healthy choices

• Goal 3: Provide family engagement, development, and education strategies

• Goal 4: Utilize a continuous quality improvement program environment

The purpose of this report is to provide formative and summative feedback to personnel

at the BGCSC and OSPI regarding the third year of the 21 st Century Community

Learning Centers (21 st CCLCs). Although summative in nature regarding the impact of

the program on the staff, parents, and students involved, the report is also designed to

provide formative feedback to program leaders. The report begins by providing general

information about the goals and objectives of the program. The introduction is followed

by a description of evaluation activities, evaluation findings, and recommendations.

Logic Model

2

THE BERC GROUP


Beginning with the 2019-2020 school year, each program site worked to create a Logic

Model to support their vision and help drive decision making. Program leaders and site

staff worked with the external evaluator to discuss the current and future needs of

students and community members. An example of Concrete’s 2020-2021 logic model is

included in Appendix A. This process will be conducted with evaluation teams for all

future years of the grant.

Evaluation Design

Researchers used a multiple measure, mixed methodology approach to conduct this

evaluation. The collection of both quantitative and qualitative data adds scope and

breadth to the study in addition to providing the ability to triangulate findings (Creswell,

1994). During the 2020-2021 school year, school closures and COVID restrictions

impacted the ability to collect quantitative data from program sites, as state assessments

were cancelled. As a result, the Year 3 report will look different than prior and future

reports. Program leaders were asked to provide their perceptions of the impact of COVID

on programming and student outcomes. Those perspectives are included in the evaluation

narrative.

Despite school closures, researchers followed a utilization-focused model of evaluation.

This model of evaluation is intended to provide useful feedback that can be implemented

in real time (Patton, 2013). As such, The BERC Group worked closely with program

leaders to develop evaluation questions and design. Qualitative data was gathered during

program observations, interviews with staff, and document review. The Boys and Girls

Club provided a database of student attendance for analysis by researchers.

Evaluation Questions

To address program goals and grant objectives, researchers collaborated with the program

leader to assure alignment of evaluation questions with the goals and requirements of the

grant. The evaluation questions were designed to provide program leaders formative

feedback for continuous improvement, as well as summative feedback about progress

toward grant goals. The questions align with OSPI’s local evaluation guidelines for 21 st

CCLC programs. The evaluation questions are:

1. What strategies and activities support project goals?

THE BERC GROUP

3


2. What contextual factors influence program implementation?

3. To what extent does the program reach the target population?

4. To what extent do students persist in the program?

5. Outcome Questions:

a. To what extent does the program contribute to improved math and reading

scores?

b. To what extent is attendance at the program correlated with academic

achievement?

c. To what extent does the program enrich students’ learning experiences?

d. To what extent do families engage with the program?

As a result of COVID school closures, question 5 parts a and b were not addressed during

this Year 3 evaluation. Researchers did ask program participants to share their

perspectives of student outcomes during the 2020-2021 school year. These perspectives

are discussed throughout the report.

Data Sources

To address these questions researchers gathered data from multiple sources throughout

the evaluation cycle. BGCSC provided a database of student attendance. Additionally,

BERC researchers conducted

• Program interviews with staff and leadership

• Program observation

• A review of initiative documents and materials

Overall Findings

EQ1. What strategies and activities support project goals?

During Year 3, the Concrete 21 st CCLC met on-site, utilizing a 1:10 staff to student ratio

to cohort students. Compared to prior years, attendance was low, with approximately

8-12 students participating regularly between the elementary and secondary cohorts. Both

groups met in the BGCSC trailer, utilizing separate rooms. The program director shared

that they were trying to maintain consistency with prior years, but the poor attendance

4

THE BERC GROUP


made that challenging. One staff member commented, “We were getting 1 kid a day on

the elementary side, [and] a few on the high school side, so our big push has been to try

to bring more kids in, in a safe way. A lot of this has come down to reaching out to

families…Bringing those kids in really breathed new life into the club. One kid is fine,

but it is not really a club environment.” He continued to note that as a few more students

engaged, they were able to offer more enrichment and academic programming.

Program leaders and staff expressed pride in their programming this year, noting that they

focused on Social Emotional Learning (SEL) to meet their goals around student behavior,

family connection, and academic support. Students, they proposed, were struggling to

participate in school, and needed more support and guidance on how to re-acclimate to

social and academic pressures and expectations. Once attendance increased slightly,

program staff offered a range of activities, keeping in mind the restrictions in place for

social distancing. One staff member noted, “I believe [SEL] is a strong foundation for

academics, and has been more of a focus this year because of COVID. [Our] kids coming

to club after hours and hours on the computer was rough. Our ability to help those kids

build coping mechanisms was huge. COVID gave us an excuse to really focus on it, and I

hope we can keep it going, since we have had great results.”

Program offerings included Power Hour, which did not look “traditional,” but staff felt

was critical to building rapport and relationships with students. Staff also noted that

students were really struggling with their academics, and seemed to benefit from the

individualized attention that 21 st CCLC staff were able to provide due to the low

attendance numbers. In addition to Power Hour, program staff partnered with the Farm to

School program, which provided experiences for kids to engage in hands on learning.

Students got to spend time outside in their garden, learning to farm, and incorporating

plant learning and nature into their activities and experiments. One staff member also

discussed their reading buddy program, which matched older students with an elementary

partner to improve reading skills. He shared,

We also have been trying to balance the needs of our different ages. Our two

Kindergarten students were struggling with reading, so we started to have some of

the older kids act as buddies for the littles, helping them to read. This was part of

the SEL work, but it was also academic support and leadership opportunities for

the older students. The older students loved it. Giving them roles made them feel

important, they felt needed, and they were honestly doing important things. They

enjoyed getting to bond, and feel important.

THE BERC GROUP

5


In response to COVID and overall student well-being, the 21 st CCLC staff introduced a

Healthy Habits program with their students, as hygiene and personal space became such

an impactful part of the club experience. Staff conducted a series of lessons focused on

hygiene using the Healthy Habits curriculum. One staff commented, “We gave them the

space to know that beyond all of the COVID restrictions, [there are hygiene aspects] we

do every day. The kids responded pretty positively.” Staff also spoke highly of the

organized and thorough COVID protocols that were in place, making it possible for

students to participate safely throughout the year without being anxious about getting

sick.

EQ2. What contextual factors influence program implementation?

During Year 3, factors that impacted the ability for program to operate at capacity

included issues around staffing, transportation, technology, and COVID restrictions.

There were several staffing challenges during the 2020-2021 school year that impacted

programming. Roles within the BGCSC changed, and leadership of the 21 st CCLC shifted

accordingly. Additionally, it was a challenge to find staff interested in working in the

program, due to location and salary. The program director noted, “[Our] location makes

staffing an issue. Also, and just a speculation, but with unemployment being so generous,

we are all having trouble getting staff, since we can hire just above minimum wage.”

Program staff also commented on challenges around staffing, however, they felt that they

had been able to experience success as a team. One staff member shared,

We work well together. We have been able to build up our club culture, and create

an environment we want to work in. In some ways starting from the ground up

has been really positive. We have a really great starting point with great attitudes.

[Staff] are rolling with the punches…we have been flexible and understanding of

the complicated time we are in. Even though we are a small team, we run

ourselves effectively and confidently.

In addition to challenges with hiring and retaining staff, the lack of transportation

available to students was a significant barrier throughout the year. Students who were not

in school did not have a way to participate. The program director commented, “With

transportation, we would have had more kids and more robust programming.”

Technology use was another challenge during Year 3. Program staff expressed their

concerns about the impact of so many hours in front of computer screens during the

pandemic, noting effects on students’ social, emotional, and academic well-being. One

staff member shared, “We need to get kids unplugged, [and] get them outside to get some

6

THE BERC GROUP


exercise. A lot of these kids are sitting around for hours and hours every day. We are

really focusing on getting outside as much as we can, and there is some push back since

some of them like to be on their computers. [But] when the kids are outside, they are

being creative.”

Finally, the pandemic created new challenges to implementing 21 st CCLC programming.

Staff recognized the time and resources dedicated to complying with COVID policies and

procedures. They also acknowledged that it required creative thinking to develop and

implement COVID-safe programming. Staff shared several examples of games they

played during the year, including pool noodle tag, Ship to Shore, and Red-Light Green

Light. The program director shared, “some games were familiar, but they were all created

fresh since the old challenges weren’t COVID compliant.”

EQ3. To what extent does the program reach the target population?

The BGCSC 21 st CCLCs were intentionally placed in this community to provide social,

emotional, and academic enrichment to as many students as possible throughout the

service area. Since approximately 80% of the students in Concrete School District receive

free/ reduced lunch benefits, program leaders knew they would be serving their target

population. During Year 3, program staff were aware of the unique challenges impacting

their students. Although typically a robust program, it was difficult to maintain a steady

and strong attendance this year, due to COVID, transportation issues, and capacity of

space and staffing. The program continued to make an effort to reach their target

population, however, strengthening their partnership with the school district, and

maintaining their social capital in the community.

EQ4. To what extent do students persist in the program?

Researchers analyzed attendance data to better understand student engagement and

persistence in the 21 st CCLCs during Year 3. Figure 1 illustrates the average days of

attendance throughout the school year for students who were present for at least one day

of the program. At the elementary site, students attended an average of 4 days while at the

secondary site students attended an average of 2 days. It should be noted that the low

attendance is a result of the ongoing pandemic, greatly impacting the overall average

days attended. When disaggregated by month, July and August had higher rates of

attendance at both the elementary and secondary sites, with a harsh dip in the winter and

a slight increase in the spring.

THE BERC GROUP

7


Average Days of A-endence

Elementary

Secondary

53

49

32

30

4

2

Figure 1

2018-19 2019-20 2020-21

Students attending at least 30 days of the program throughout the year were given awards

and identified as ‘21 st Century Students.’ The number of 21 st Century students at each site

are displayed in Figure 2.

70

Number of 21st Century Students

59

Elementary

76

Secondary

17

7

4

Figure 2

2018-19 2019-20 2020-21

Figure 3 illustrates the percent of 21 st Century students among the total number of

program participants. Again, students were considered in the total if they attended at least

8

THE BERC GROUP


1 day of the program in the Spring or Fall session. Of the total counted, 32% of

elementary students earned 21 st Century status and 40% of secondary students earned 21 st

Century status.

Percentage of 21st Century Students

Elementary

Secondary

57%

61%

33%

31% 32%

40%

Figure 3

EQ5A. To what extent does the program contribute to improved math and reading

scores? Due to the suspension of state standardized tests and distance learning for most

of the 2020-2021 school year, researchers were unable to collect state assessment scores.

This research question will be revisited once assessments resume, hopefully in 2022.

Youth Program Quality Assessment

2018-19 2019-20 2020-21

To fulfill grant requirements, researchers used the SA-PQA (School-Aged Program

Quality Assessment) and YPQA (Youth Program Quality Assessment) tools to evaluate

the quality of youth experiences in the program. These assessments measure four main

domains of programming, including safe environment, supportive environment,

interaction, and engagement. Youth-centered policies and practices, high expectations for

youth and staff, and access were other areas incorporated in the tool. The YPQA was

developed by the Weikart Center, and according to their website (2014), “each domain

contains items that focus on specific elements of best practice.” The findings provide

opportunities for staff members to reflect on what they are doing well, to determine ways

they can improve the program, and “to build professional competencies.”

Figure 4 and displays the overall domain scores for the self-assessments conducted at

Concrete High School during Year 1 (2018-2019) and Year 2 (2019-2020). Programs did

not participate in the YPQ assessments during Year 3, but will resume in Year 4.

Self-assessments are conducted internally by trained program staff. The scores reflect

THE BERC GROUP

9


observations from the Power Hour, Positive Action, and Smart Girls learning experiences

during Year 1, and Positive Action and art experiences during Year 2. Across all domains,

self-assessment scores in Year 2 decreased.

Concrete High School YPQ Self Assessment Over Time

4.6

4.53

4.23

3.9

3.08

2.88

3.67

2.67

Figure 5

Safe Environment Supportive Environment Interaction Engagement

2019

2020

Figure 5 shows two years of scores for external assessments, conducted by BERC

evaluators at Concrete High School in December 2019. External assessor scores for

Supportive Environment and Interaction were lower in 2020, while scores in Safe

Environment remained consistent, and scores in Engagement increased. Scores in

engagement have been the lowest across years and between external and selfassessments.

Concrete High School YPQ External Assessment Over

Time

4.9

4.83

4.5

3.74

4.02

3.15

2.17

2.33

Safe Environment Supportive Environment Interaction Engagement

Figure 6

2019

2020

10

THE BERC GROUP


In a similar trend to Year 1, several individual items scored low (below 3.0 out of 5.0) on

both assessments, including opportunities for students to collaborate, develop their

leadership capacity, make choices about their learning, and reflect on their experiences.

Recommendations from The Center for Youth Program Quality included a focus on the

domains of Interaction and Engagement:

o

Cooperative Learning: Providing young people an opportunity to

participate in and lead small groups has a positive impact on classroom

climate, self-esteem among students, internal locus of control, and time on

task. Students in cooperative teams are more active, self-directing, and

expressive, all of which may be associated with achievement gains.

o

Youth voice: Providing young people with chances to make decisions

about their activities and how they carry them out can improve motivation

and buy-in, and more importantly, offering choices in a youth program

space gives youth a chance to practice for the bigger choices they'll make

outside of the program.

o

Ask, listen, encourage: Positive relationships can open the gateway to

learning. Studies find that relationships with "warmth, connectedness,

good communication and support" aid in positive youth development and

are connected to academic success (National Research Council and the

Institute of Medicine, 2002). Asking effective questions, listening to

youth, and encouraging youth through positive and specific feedback can

help to build positive relationships and may influence the intrinsic

motivation of youth.

Figure 6 displays the overall domain scores for the self-assessments conducted at

Concrete Elementary School in December 2019. Scores reflect observations from the

Power Hour, Triple Play, Lego Robotics, Project Read, and Art learning experiences.

Scores in 2019 were lower in all four domains, with the greatest difference in

Engagement.

THE BERC GROUP

11


Concrete Elementary School SA-PQ Self Assessment Over

Time

4.07

3.84

3.79

3.56 3.45

3.46

3

2.08

Figure 7

Safe Environment Supportive Environment Interaction Engagement

2019

2020

External assessor scores for the December 2019 program visits were higher than during

Year 1 in all domains except Safe Environment, which decreased only slightly between

years (Figure 7). Additionally, external assessor scores were higher than self-assessment

scores for each domain.

Concrete Elementary School SA-PQ External Assessment

Over Time

4.9

4.79

4.25

4.41

3.37

3.28

2.42

2.54

Safe Environment Supportive Environment Interaction Engagement

Figure 8

2019

2020

Opportunities for growth identified by both self and external assessments include creating

more opportunities for students to have voice and choice in their learning, incorporating

12

THE BERC GROUP


reflection, and helping students to develop skills to plan activities around their learning.

The Center for Youth Program Quality provided the following recommendations to

support further program growth:

o

Youth voice: Providing young people with chances to make decisions

about their activities and how they carry them out can improve motivation

and buy-in, and more importantly, offering choices in a youth program

space gives youth a chance to practice for the bigger choices they'll make

outside of the program.

o

The skills of making plans for the future and learning from the past can

help youth succeed in school and in life. These skills are tied into what

brain scientists call executive functions, and play an important role in

directing attention to tasks and decision making that connects with

consequences.

Conclusions

Over the past three years, the Concrete school district and Boys & Girls Club have

developed a sustained and committed partnership to serve students through their 21 st

CCLCs. Students have benefit from programming that supports their social, emotional,

and academic needs, while also attempting to engage them in critical and creative

thinking, community action, and persistence through difficulties.

Concrete faced challenges this year, including staff retention, COVID restrictions, and

limits to transportation resulting in poor attendance. Despite these challenges, staff shared

that they felt proud of their work, and were able to develop meaningful relationships with

students. Program leaders also shared that they were excited for their summer program,

which would look different due to COVID, but provide them with unique opportunities to

support students more holistically. One program staff member shared, “One positive for

us is that since Concrete is doing an in-depth summer school program, we can work

closely with them to help kids catch up a little, and get ready for next year. We will get to

work with the teachers, and see where the kids are at, and where they are struggling the

most.”

THE BERC GROUP

13


Recommendations

Year 3 results provided evidence to support the value of the program in helping students

to make build relationships with adults, and develop important social emotional skills

needed to be successful in school. To continue to make improvements, we suggest the

following recommendations, which are a continuation of a more typical year of

programming:

Continue to build capacity for community and family partnerships.

A strong focus of the program has been cultivating positive partnerships with community

and family members. These partnerships should continue to be sought out and built upon

in the future, as they result in more positive experiences for students. One

recommendation is to expand the scope of partnerships to increasing volunteer

opportunities and service-learning projects, giving students a chance to explore their

interests in the community. Allowing student voice and choice is aligned with the

program mission and vision and will likely increase student engagement while providing

valuable, action-oriented experiences for students.

Increase student reflection in projects.

Results from the YPQA suggest that program leaders should increase opportunities for

student reflection into regular programming. Recommendations from the YPQA notes

“The skills of making plans for the future and learning from the past can help youth

succeed in school and life. These skills are tied into what brain scientists call executive

functions and play an important role in directing attention to task and decision making

that connects with consequences” (YPQA, 2017). Providing dedicated time, at the start or

end of an activity, gives students the opportunity to regularly reflect on their learning

throughout the program. These reflections can be verbal, written, or even illustrative, and

should include sharing their thoughts with peers. For example, students can be given exit

slips at the end of each program session that ask reflection questions such as “Today I

Learned….” Program leaders can also use this information to improve the student

experience. In light of current pandemic-based restrictions, these exit slips, or any

reflections, can be given online, through Google or Microsoft Forms.

Collect and Use Data

14

THE BERC GROUP


Program leaders at each site should continue to document and collect data required for

the program and the evaluation. Additional training on the YQPA, as well as regular time

to perform and reflect on the tool should be allocated. This data provides critical, ongoing

information on student engagement and the effectiveness of enrichment activities, which

can be used to inform and improve the 21 st CCLCs. Additionally, the use of real time

student data will help program staff to better meet the needs of students. The sharing of

data with the school, family and outer community members may help to create a strong

system of support, awareness, and recognition for the program.

Continued Focus on Program Activities and Expectations that Align with the Vision

It is important that program staff members continue to plan activities early in the year, in

alignment with the vision and goals of the program. Program leaders should continue to

find innovative ways to communicate goals and activities to general education teachers to

ensure all stakeholders are operating under the same beliefs. One suggestion is to share

the program’s logic model with all stakeholders, so they can see and have discussions

about the best ways to collaborate for the benefit of the students.

THE BERC GROUP

15


Appendix A

2019-2020 Logic Model

16

THE BERC GROUP


References

David P.Weikart Center for Youth Program Quality. Youth Program Quality Assessment

and School-Age Program Quality Assessment. Obtained 8/27/2014 from http://

www.cypq.org/assessment

THE BERC GROUP

17


The BERC Group, Inc.

P.O. Box 3552

Redmond, WA 98052

Phone: 425.486.3100

Web: www.bercgroup.com

18

THE BERC GROUP

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!