Lessons from the Field - Seer Consulting

Evaluation of Capacity Building: Lessons from the Field
Part III: Important Considerations and Lessons Learned
20
Contextualized
Capacity-building agents vary (management
support organizations, independent consultants,
intermediaries, foundations) as do their strategies.
Their communities and constituents are even
more diverse.
A hospital is different from a charter school is
different from a performing arts group is different
from a coalition of providers that works with homeless
people. Rural is different from urban.
Organizational life cycles require different leadership
styles and management approaches.
Interviewees stressed the need for evaluators (or
evaluation processes) to understand the context in
which the nonprofit or project is situated—what
the norms are within its field, the culture of the
work, the political and socio-economic impacts.
Measurement tools may be the same across a field,
but it is defining and answering critical questions
that get at the underlying reasons of why and how
an intervention worked or did not work. For
instance, were collaborations formed or enhanced
as a basis for ongoing change? Qualitative
approaches within the evaluation will often be more
useful in understanding the contextual issues within
which capacity building is taking place.
Pluralist or culturally based approaches offer an
emerging area for understanding the work, defining
potential versus realized impact, and informing evaluation
methods. The insights of Donna Mertens and
others point to the roles that culture and politics
can play in evaluation work and the program development
or implementation that becomes the focus
of evaluation. Mertens acknowledges that “the
direct relationship between the evaluation of social
and educational programs and access to resources
sets the stage for conflict,” 12 giving rise to pluralist
concepts “in which multiple methods, measures,
criteria, perspectives, audiences, and interests
were recognized.” 13
In particular, Mertens 14 points to emergent
approaches such as the:
• Emancipatory paradigm (raising the question of
what social justice and fairness mean in program
evaluation);
• Empowerment evaluation (as a means to foster
self-determination, generate illumination, actualize
liberation and institutionalize systematic
evaluation);
• Developmental evaluation (matching a program’s
developmental character with evaluation that is
developmental—i.e., constantly searching for
ways to be responsive to an ever-changing set of
conditions); and
• Participatory evaluation (process that involves the
stakeholders in various tasks of the evaluation so
that the results are fully comprehensible to
participants).
Customized
If an organization defines a theory of change,
completes an assessment and describes contextual
issues (influencing factors) from the outset, then it
is essential to customize an evaluation that reviews
all these pieces both separately and together.
If evaluation is to take into account a specific
theory of change as well as contextual issues, it
must be customized. The evaluation must ask critical
questions that pertain to the project, including
the development of surveys and other instruments
that tie into the strategies being used in the intervention
and then adapting tools as needed. When
starting its evaluation of groups provided with technical
assistance funding, for example, the Hawaii
Community Foundation at first used a well-known
assessment tool and later customized it to work
with and for the Hawaiian nonprofit community.
Learning-based
The ultimate purpose of evaluation should be
focused on continuous learning and developing
practices that move organizations toward greater
effectiveness.
Paul Light ends the research-based Pathways to
Nonprofit Excellence by writing:
Nonprofit organizations vary greatly in their
characteristics and strategies in taking the
first step toward higher performance. Some