Lessons from the Field - Seer Consulting

Evaluation of Capacity Building: Lessons from the Field
Part IV: Case Studies
38
What is nonprofit capacity building?
Borrowing from Grantmakers for Effective
Organizations, OMB staff members say, “We could
easily define capacity building as ‘increasing the
ability of an organization to fulfill its mission.’
…Although we can define ‘nonprofit capacity
building’ by staying at that high level of abstraction,
we can only measure it by drilling down to a more
concrete level. When we do so, we immediately
confront both the diversity of the term ‘nonprofit’
and the diversity of the term ‘capacity building.’
What this means is that there is not going to be one
answer to the question of how to measure the effectiveness
of nonprofit capacity building. There are going to be a
great many answers to a great many very different
concrete situations.”
Measurement: “If capacity building is an increase
in the ability of an organization to fulfill its mission,
then measuring it directly would require having a
unit of measure of this ability, something we can’t
expect anytime soon” (OMG Internal Issue Paper).
Instead, we look for improvement in various aspects
of the organizational performance. While it may be
simple to assess fundraising success because of
existing indicators (amount raised, average size of
gift, cost of raising funds and so forth), no such
metrics exist for many important aspects of organizational
performance such as strategic planning or
governance.
Organizational capacity has to be person-carried
or it is dead, yet it has to be institutionalized in
the system or it evaporates.
Goals: Research has shown that people do not
improve their performance unless they set a goal. In
regard to design of capacity-building interventions,
OMG staff will inquire, “How many cases are there
in which the target agencies set explicit improvement
goals?” This is not something that is
emphasized in capacity-building literature. OMG,
speaking from its experience in evaluating projects
where multiple stakeholders (staff and board,
consultant, funder) had different priorities, suggests
that, “Such a lack of goal alignment usually spells
trouble for the capacity-building project, but also
raises an evaluation question: Against whose goals is
the effectiveness of capacity building to be measured?”
Timelines: As with nearly every informant for this
study, OMG concurs that it takes multiple years to
build capacity. And yet nonprofits often trap themselves
by promising five-year outcomes for two-year
initiatives or 10-year outcomes for five-year initiatives.
Most evaluations do not extend beyond the life
of an initiative—making it difficult to capture the
real outcomes that occur later as a result of capacity
building. “After we factor in the first year it took for
the initiative to get underway, and the fact that evaluation
results are wanted early for re-funding, we’re
usually looking at two and a half years of effort
toward 10-year goals and wondering why we aren’t
more effective at capacity building. An evaluation
that stops measuring at the point when the intervention
stops may show no measured improvement in
capacity when in fact it simply hasn’t had time to
occur” (OMG Internal Issue Paper).
People vs. Systems: Capacity building should focus
on both people and systems. “Organizational
capacity must partake of both: It has to be personcarried
or it is dead, yet it has to be institutionalized
in the system or it evaporates….When we are measuring
the effectiveness of capacity building (or
performing an organizational assessment), we have
to look at people, systems, and how they relate and
reinforce each other….It is much easier to document
whether a system exists than to assess what
people can do, and harder still to assess a relationship
between the two” (OMG Internal Issue Paper).
The danger is that evaluation of capacity building
will be satisfied with evaluating the two independently,
or worse, focusing on only one or the other.
Behavior change and internalizing the
learning from the change: Evaluators may tend
to rate observable change. Behavior changes in the
short term can be empty if not coupled with internalizing
the learning that institutionalizes the
change. If behavior change is the indicator, it is
important to look at the long-term outcomes associ-