Partnership Evaluation Report (2020-2021)

Evaluating Ourselves: What We Learned
A Summary of Findings from the First Year of our Partnership Evaluation
(September 1, 2020 to August 31, 2021)
In 2019, the Theory of Change Working Group was tasked by the Steering Committee to develop a dynamic Theory of Change
framework (abbreviated illustration on next page) as a foundation for evaluating the ECN's partnership engagement,
scholarship, and impact. This Theory of Change has guided the Strategic Learning Team in building and implementing an
evaluation plan. This report is a summary of evaluation findings from our pilot Year 1 of this evaluation.
Authors:
Andrea Antoniuk, Judith Walton, Rich Janzen, Shelly Jun, Pieter de Vos, and Reem Razzouk
We would also like to acknowledge the individuals who were involved in the co-creation and/or implementation of the evaluation:
Victoria Dickenson, Lauren Johanessen, Ann Marie Castleman, and Susan Hopkins
ECN Partnership Evaluation Report 2021 | 1

Early Childhood Practices, Programs, and Policies
A Partnership
Theory of Change
Taking part in
our projects
leads to...
ENGAGEMENT
KNOWLEDGE &
SKILLS CREATED
ACTION &
IMPACT
LEGEND
Research
Questions
How will we
get there?
GUIDING
PRINCIPLES
1. Community Driven
2. Culturally Responsive
3. Equitable Participation
4. Action-oriented
OUR GOAL
To develop community-driven and
culturally responsive approaches for
producing and using evidence to improve
early childhood practice, programs, and
policy towards system effectiveness.
OUR OBJECTIVES
Objective #1
Deepen understanding
through engagement
and collaboration to
ensure that evaluations
are community- driven
and culturally
responsive.
How do we ensure that the demand
for, creation, and use of evidence are
community-driven and culturally
responsive?
Greater capacity
in ensuring that
research
collaboration is
communitydriven
Objective #2
Co-create and
implement research
and evaluation that
is mutually
beneficial to all ECN
partners.
Building authentic
partnerships and
collaborations
Greater capacity
in ensuring
equitable
participation of
stakeholders in
research
collaboration
Objective #3
Develop an Evaluation Hub
that connects people to
evaluation resources, learning,
and support to mobilizes
knowledge and build
evaluation capacity in the early
childhood field.
How do we increase knowledge,
skills, and motivation to create
& use meaningful evidence
through innovative capacity
building approaches?
Training and
mentoring
Greater
capacity for
ongoing colearning
Greater
capacity in
using evidence
Partners
Research and
evaluation
projects
Greater
capacity in
creating
quality/useful
evidence
ECN
Increased access to and use of
evaluation resources
Lasting connections
Deepened community of practice
How do we increase the value
and use of evidence for decision
making to improve practice,
programs, and policy?
Greater
mobilization of
people
Evaluation Field
Increased
understanding of
community-driven
and culturally
responsive evaluation
Knowledge
mobilization
Greater
mobilization of
evidence
Greater value
and use of
evidence in
decision-making
across
stakeholders
Early Years Sector
Increased culture of
learning and use of
evidence
Improved outcomes
in early childhood
Activities
Outcomes
ULTIMATE
OUTCOMES
More innovative early
childhood practices,
programs, and
policies
Greater system
effectiveness in
improving early
childhood outcomes
ECN Partnership Evaluation Report 2021 | 2

PARTNERSHIP EVALUATION: YEAR 1
1
Purpose
The purpose of this evaluation is to:
Examine how we are working according to our
four guiding principles (community-driven, culturally
responsive, equitable participation, action-oriented).
Understand the impact we are collectively making.
This is in order to learn, grow, and adapt what we do (within the
partnership) and share lessons learned with external
stakeholders (beyond our partnership).
Evaluation Questions
To what extent are our partnership’s activities (engaging
communities, co-creating/co-learning, and EvalHub
capacity building) aligned to our four guiding principles
(community-driven, culturally responsive, equitable
participation, action-oriented)?
Data Sources
Student Reflective Diaries
Comprised of 4 open-ended questions submitted online.
Students responded using text, pictures, or mind maps (see the appendix for
sample mind maps created by students!).
Data collection: December 2020 - August 2021 (n = 15)
Meeting Minutes
Reviewed meeting minutes from all ECN working groups and committees to
triangulate themes.
Data collection: September 2020 - August 2021
Post-Meeting Surveys
Responses were submitted online after Knowledge Exchange Working Group
and Steering Committee meetings.
Data collection: September 2020 - August 2021 (n = 27 responses).
2
3
4
How, and to what extent, are we effectively building
capacity both internally as we equip ourselves to do our
work, and externally as we mobilize what we are learning
with others?
How, and to what extent, is our partnership’s work
reaching our ultimate goal to improve early childhood
practice, programs, and policy towards system
effectiveness? How are our collective efforts contributing
to the development of children in Canada?
What can we do to improve the effectiveness of our
partnership in meeting our desired outcomes? What are
we learning about how to carry out such a partnership?
Researchers (14.81%) Leaders/Managers (7.41%)
Funder (1.85%) Students (24.07%) Academics (42.59%)
Private Sector (1.85%) Non-pro t Sector (7.41%)
ECN Partnership Evaluation Report 2021 | 3

Evaluation question 1: To what extent are our partnership's activities (engaging communities, co-creating/co-learning, and EvalHub capacity
building) aligned to our four guiding principles (community-driven, culturally responsive, equitable participation, action-oriented)?
Community-driven &
Culturally Responsive
Respondents strongly agreed about the need to build a shared understanding of community-driven
and culturally responsive principles. Literature reviews are one way to begin understanding how
these concepts and principles are articulated; consequently, we are conducting a series of scoping
reviews to unpack some of these principles.
Equitable Participation
There was strong agreement that efforts in engagement and collaboration
promoted equitable participation. Respondents felt that they were:
Action Orientation
Respondents mentioned that having structure helped
to facilitate productive meetings, progress towards goals, and
remain action-oriented. Structure was achieved through:
...able to speak their
minds and be heard.
...able to give input that
contributed directly to
progress.
pre-work
before
meetings
...actively engaged in
providing input.
... especially engaged when they
were put in breakout rooms for
discussions.
...in an inclusive
environment where
students and new
attendees were welcome
to give input.
goals and
timelines
meeting
agendas
having difficulty participating when they had to transition
in and out of projects at different stages of progress.
shared documents and slide decks
ECN Partnership Evaluation Report 2021 | 4

Evaluation question 2: How and to what extent are we effectively building capacity both internally as we equip ourselves to do our work,
and externally as we mobilize what we are learning with others?
Capacity Building
Respondents identified that we are building capacity by:
Transferability
Students agreed that through their roles within the ECN, they are
building capacity in the following areas:
adapting
to
working
remotely.
figuring out
what we can
do as a
partnership.
reaching a
common
understanding of
how to engage
stakeholders.
academic skills
evaluation skills
stakeholder
engagement skills
collaborating
in new ways
(e.g., shared
documents,
breakout
rooms).
Graphic submitted in a
student reflective diary
We recognize that these findings predominantly reflect how we
are building capacity internally (i.e., equipping ourselves to do our
work). Forthcoming years of our partnership evaluation will focus
on both our internal and external capacity building.
There were mixed responses about the perceived impact of
the skills gained. While some students expressed uncertainty that
their skills will transfer outside of academia, others felt confident
they would be able to apply the skills they have learned to
support early childhood development. Many reported a new
or renewed interest in community-based evaluation and research.
ECN Partnership Evaluation Report 2021 | 5

Evaluation question 3: How and to what extent is our partnership's work reaching our ultimate goal to improve early childhood practice,
programs, and policy towards system effectiveness? How are our collective efforts contributing to the development of children in Canada?
What Students Are Saying...
Some students are not certain how their work
contributes to the broader ECN initiative.
Other students found that the larger network
meetings help them to see where they fit in.
Progress
There is an overall sense of progress towards our
ultimate goal, due to building foundations in terms of
capacity building, communications, and shared
understandings. Specifically, respondents mentioned:
connecting with the
larger network
building general capacity
by sharing knowledge and
information
clearer communications
(e.g., website content)
building a shared understanding of
our goals and values
ECN Partnership Evaluation Report 2021 | 6

Evaluation question 4: What can we do to improve the effectiveness of our partnership in meeting our desired outcomes? What are we
learning about how to carry out such a partnership?
Engagement
Intentionality
There was strong agreement that we can further
improve our effectiveness by increasing
accessibility, connections, and clarification.
Respondents suggested:
Multiple means for contributing to projects
and accessing information.
Connecting with more individuals and
organizations from the early childhood field.
Obtaining feedback from outside of our
"bubble".
Defining and clarifying engagement in terms
of what we are inviting people to do.
Some students specifically asked for more:
Opportunities to apply their skills in
community settings.
Methods of connecting with others in the
network (e.g., students, community groups).
Updates about the work that everyone is
doing to progress towards our ultimate
outcome.
Structure
Recommendations for
maintaining structure, which
respondents felt was beneficial to
productivity and progress,
included:
meeting agendas
strategies for when members cannot
attend meetings
more meeting updates on the
progress of the network
clearer expectations, goals, and task
lists for students
There was strong consensus about the value
of using language and communications more
intentionally. Respondents suggested:
Building a working
group for systematic
communication and
knowledge
mobilization.
Reaching a shared
understanding of
community-driven and
culturally responsive
practices.
Using communications
to show the
connections between
students and the
partnership.
ECN Partnership Evaluation Report 2021 | 7

CONCLUSIONs
In the first year of our partnership evaluation....
What have we done well?
We created strong foundations that include engagement and equitable
participation, capacity building, and connecting with the larger network.
We utilized an action orientation that was supported by strategies for
promoting and maintaining structure.
Join us next year
as we keep learning
together!
What are some areas for growth?
Continue to promote engagement, especially among students and
those who work in the early childhood development field.
Continue to build structure to maintain and increase effectiveness.
Reach shared understandings about key concepts such as communitydrive
and culturally responsive practices.
Improve communication across the partnership.
ECN Partnership Evaluation Report 2021 | 8

APPENDIX: STUDENT REFLECTIVE DIARY ENTRY SAMPLE
Networking
ECN Partnership Evaluation Report 2021 | 9

APPENDIX: STUDENT REFLECTIVE DIARY ENTRY SAMPLE
ECN Partnership Evaluation Report 2021 | 10