Chapter 6 - Baobab

BAOBAB: Handbook on Implementation Management – 6. Evaluation 1
6. Evaluation
6.1 What to expect in this chapter? An overview of topics
The Core issues part of this chapter covers:
• a definition of evaluation, and the difference between an evaluation
from within or from the outside
• how these types of evaluation fit into the implementation process
• choices in evaluation: performance evaluation and impact evaluation
• who we do evaluations for, and,
• who does them.
Under tools, internal evaluation (self-evaluation, SWOT analysis) and external
evaluation (questions around commissioning an evaluation, participatory
evaluation and working with an evaluator) tools are presented.
6.2 Core issues
6.2.1 What is an evaluation ?
Basically, evaluation is about assessing
what is (the actual /
current situation)
in relation to
what was intended (the
plans, targets, objectives)
The process becomes complex when one has to consider
different expectations, agendas, objectives and the related achievements,
the lapse of time between objectives setting and achievement (or: how over
ambitious one might have been when setting objectives),
and the different views of reality (past and present) of different people
involved.
Because of this complexity, methods for evaluating have been designed to
organise and manage the evaluation process, so that it is understood and
transparent (and is then acceptable), and useful to all those concerned.
Usefulness is important, as evaluation is essentially a management tool, a
mechanism for learning about a project, programme, an enterprise or institution:

BAOBAB: Handbook on Implementation Management – 6. Evaluation 2
Are you doing things right?
(for example: with respect to your intentions or plans)
Are you doing the right thing?
(for example: in terms of real impact, policy and current
thinking)
The information gathered to answer these basic questions assists in decisionmaking
about the ongoing management of a project, its life or its closure. The
process can be considered “diagnostic and therapeutic”: it can result in changing
activities, re-allocating resources, re-planning, shifting focus, expanding or
rightsizing operations, organisational mergers, or even in closing down
altogether.
A definition to consider...
An evaluation is a process to enable a
project to examine its effectiveness
and efficiency after a reasonable
period of implementation.
The purpose of an evaluation is to
affirm the project’s work (plans,
activities, impact), or to transform
it, so that the project can
effectively deliver, make an impact
and meet the challenges it faces.
Questions:
What is effectiveness?
What is efficiency?
What does effectiveness
translate into in your project?
What does efficiency translate
into in your project?

BAOBAB: Handbook on Implementation Management – 6. Evaluation 3
6.2.2. Evaluation and project cycle management
An evaluation event is distinct from the ongoing monitoring of activities, resources, outputs, and even impact of a project.
Monitoring involves gathering information for day-to-day decision-making and management. Evaluation focuses on in-depth
reflection at a switch-point considered significant in various life stages of a project; (this could be at the end of a work
phase, whenever special circumstances demand it, for example a crisis), or at the end of a project itself.
Evaluation asks: How have we done? after a project period is completed
Monitoring asks: How are we doing? whilst a project phase is running
inform plan motivate steer inform
project
idea
Identification Phase
set objectives
decide
• understand actual
situation
− problems, actors, views
− objectives, visions
− potentials
• establish system of
objectives
project
objectives
Design Phase
organize
control
• establish project
strategy
• agree on structural setup
− organisational
− financial
• prepare decision to
implement the project
project plan
implement
monitor
evaluate
implement
Phases of Implementation
• operationalise planning
• organise internal
structure and external
relations
• assess implementation
through Monitoring
• evaluate
monitor
evaluate
end of
project
• make required adjustmen
• report
• re-plan
• terminate project

environment
BAOBAB: Handbook on Implementation Management – 6. Evaluation 4
6.2.3. Levels of evaluation: performance and impact
We repeat something already presented in previous chapters:
Clients / customers / target
groups / beneficiaries /
members
Outputs /
Services
Activities
Resources:
Personnel – material
-finances
Inputs
Commissioning
agency / donor /
members
Task
Frame conditions
Information
Management
decisions
Strategic planning
monitoring
organising
Planning operations
replanning
E
V
A
L
U
A
T
I
N
G
impact
performance

BAOBAB: Handbook on Implementation Management – 6. Evaluation 5
Please refer to the above diagram, parts of which you will have seen in
previous chapters. On the right-hand side under “evaluation” terms
commonly used by professional evaluators are related to the project system
we have introduced here, namely evaluating cost-effectiveness, benefits in
relation to cost (cost-benefit), impact evaluation and performance
evaluation. Performance and impact evaluation are most common and any
comprehensive evaluation includes both:
Performance: are all resources being used efficiently?
• can you do better what we are doing?
• is your staff competent and efficient ?
• can you produce more / better goods?
• can you deliver more / better services?
• can you improve on quality?
Impact are the project beneficiaries using what we
deliver?
• are the intended beneficiaries benefiting, or someone
else?
• are we making the impact we set out to make?
• are our goods/ services responding to the community’s
need?
At all levels of evaluation appropriate measurement is necessary,
for example:
How much …? Over what time period? At what cost?
How many people used our services? Over what
period of time?
How many people benefited? Over short-, medium-,
and long term?
What was the impact on the benefiting community?
Was the impact different among different groups?
6.2.4 Internal and external evaluation

BAOBAB: Handbook on Implementation Management – 6. Evaluation 6
Internal evaluation involves a self-reflection or self-evaluation process, and
is conducted within a project or organisation. Project staff would take on an
internal evaluation as a step following (and based on) the monitoring of
project processes (planning, implementation, and interaction with the
external environment). It is an event during which an internal assessment of
information (gathered during monitoring) happens, focusing on progress in
implementation, in relation to plans and the project environment. It is
conducted by project management and staff.
External evaluation happens with the involvement of someone from the
outside, often as a result of an external impetus, for example from a donor.
It is usually initiated at a crucial point in the life of a project, for example
∗ when the project results and projected benefits
defined in the project plan are checked (maybe
there is evidence that they cannot be attained)
∗ when important frame conditions change (for
example qualified and experienced staff cannot be
found)
∗ if a fresh look from the outside is considered
necessary for a long-running project
∗ when a funding phase is nearing completion and the
question of re-funding (and to what extent) arises
∗ when the experiences gained in a project are to be
used to improve the implementation of other
projects or to plan new ones.
6.2.5. For whom do you evaluate?
Evaluations yield information useful to many stakeholders in projects...

BAOBAB: Handbook on Implementation Management – 6. Evaluation 7
o as you have seen, donors or funders require us to evaluate
o partner organisations sometimes require assessments of our work
o the communities we serve require reports on our work and the funding
we raise in their name
o the project staff need evaluations to see how they are doing, and how
they can improve.
Depending on for whom an evaluation is meant, the orientations and the
forms of evaluation may differ.
6.2.6. Who conducts evaluations?
Internal evaluations take place within the project as staff and
management participate in a self-reflection process without external
intervention.
External evaluations have (historically) been conducted as externally driven
evaluations or externally facilitated evaluations.
Externally driven evaluations are those arranged with external evaluators,
and / or instigated - and funded - by donors.
There are some potential strengths and some potential weaknesses built in
to this approach, which a project team may want to make use of, or against
which you may want to guard.
External evaluations are conducted by individuals or teams employed on
short term contracts to investigate the project and compile a report (for
the donor, which may be copied to project staff).

BAOBAB: Handbook on Implementation Management – 6. Evaluation 8
The evaluators often work on site, and interview project staff in depth, but
the frame, structure and process of the evaluation is decided between the
expert evaluators and the commissioning agency (e.g. the donor).
Potential
strengths:
Potential
weaknesses:
What do you
want to do?
External person(s) observe internal matters.
External (often international) expertise brought in to
comment on a project.
Evaluators can bring in new objectives or strategies not
considered before.
Evaluators cannot get a realistic picture of the project
in a short time.
Report is in the donor’s language which may differ from
the project language.
Donor’s questions may be answered in the evaluation but
not those of the project staff.
Results can be judgmental and relate to evaluators’
standards of performance or impact.
Evaluators get information but do not give back.
Prepare carefully to introduce external persons into the
project reality.
Get a full and uncensored report in the project language.
Make sure staff questions are included in the evaluation
brief.
Ensure project own objectives and strategies aare used
as reference for assessments.
Generally, therefore you may want to pursue the participatory evaluation
approach:
In externally facilitated evaluations people external to the project
facilitate a participatory process in which the project stakeholders (staff,
beneficiaries, partners) have a say in the frame, structure, scope and
process of the evaluation, in order that the evaluation results may be of

BAOBAB: Handbook on Implementation Management – 6. Evaluation 9
practical use to themselves. The external evaluators bring in external (often
international) project management and technical subject (e.g. in rural small
business) expertise, whilst ensuring that the voices (concerns, expertise,
opinions, complaints) of all local stakeholders are heard. 1 The facilitator
ensures that the process is honest and open, and is in line with appropriate
current thinking in the relevant fields.
1 This is known as a constructivist approach to evaluation as people are allowed to
construct their own evaluation criteria, to participate in information gathering and
assessment, make their own consensus judgements, and develop their own recommendations.

BAOBAB: Handbook on Implementation Management – 6. Evaluation 10
6.3 Tools
6.3.1 Internal evaluation
One helpful tool to prepare for and focus an internal self-reflection process
is the SWOT analysis, which looks at Strengths and Weaknesses that have
emerged in the past, and Opportunities and Threats anticipated for the
future. It can be conducted in a workshop situation, and/or among different
groups ensuring that all views, opinions and realities are included in the
picture.
Examples of useful questions are:
past future
positive
negative
General:
How did we respond to which
challenges?
In which areas do conflicts arise?
Strengths:
What have we done well?
What needs have we met?
What have been our strong points?
What have been our assets?
Weaknesses:
Where have we failed?
At which levels have we had problems?
What have been our weak points?
General:
Is government policy or
legislation likely to change and
affect us positively or
negatively?
Is the funding environment
likely to support our work?
Opportunities:
Who will need us, our services,
our products?
Threats:
Who else will be offering the
same or similar services?
Who else will be producing the
same products?
Will we complement one another
or are we rivals?

BAOBAB: Handbook on Implementation Management – 6. Evaluation 11
SWOT is (only) a preparatory tool as it helps to brainstorm ideas but does
not help in the assessment. A specific tool in assessment is the comparison
between ideas / targets / hopes and where you are at the point in time, the
comparison between the actual and the plan.
This can be done in the following 7-steps-sequence:
1. What did we want to achieve? What were our targets? Our hopes?
Our expectations?
2. What would the above have meant specifically for now, for the point
in time of the evaluation?
3. Where are we now? What have we really achieved? What is our actual
situation? What effects did our work have that were not foreseen?
4. Where have we actually progressed faster than we thought? And
where are we behind in our targets / hopes / expectations? Where
are we on schedule? Where are we besides it?
5. What are the reasons for good progress? What are the reasons for
being behind? What are the reasons for being on / behind schedule?
How do we understand the deviations?
6. What do we specifically learn from all of the above for each issue?
What do we conclude?
7. What do we want to change immediately? What do we want to do
differently in the medium or long term?
These seven steps should be applied to every issue in your project. Then you
will get a systematic evaluation with an intention to change from learnings of
the past. Obviously a table can be used to present the findings in a summary
result form.
1.
Targets
Issue A.:
…
Issue B.:
…
2.
Specifics
3.
Actual
situation
4.
Deviations
5.
Reasons
6.
Learning
/ Conclusions
7.
Changes

BAOBAB: Handbook on Implementation Management – 6. Evaluation 12
Issue …:
…
Issue …:
…
Issue …:
…
6.3.2 External evaluation
External expertise to facilitate the process of an evaluation is recommended
here. In this case staff, management and stakeholders of a project
participate in framing and answering the evaluation questions, assessing the
answers, and recommending appropriate changes.
A desirable procedure for this process follows. It is basically the same as
explained for the internal evaluation, only a bit more detailed. Use it as a
basis in discussions with an evaluation facilitator, and to track his/her
process.
Steps
1. Announcing the evaluation and at the same time affirming the type of
evaluation process: Who is to be informed, actively involved, and

BAOBAB: Handbook on Implementation Management – 6. Evaluation 13
how? Are the principles of participation and external facilitation
clear to all concerned?
2. Deciding on the aim of the evaluation: Why an evaluation at this
time? What are we trying to achieve? What do we want to evaluate?
What is the purpose of the evaluation? What do the various
stakeholder groups want from this evaluation?
3. Deciding what is to be evaluated: Is it important to look at
performance or impact or both? Are we looking at impact on all
groups or choosing a narrower focus? What is most useful for which
stakeholders (donors, beneficiaries, staff, managers) here?
4. Appointing the evaluation team: Who will co-ordinate, plan and
organise the evaluation? Will the evaluation be facilitated by an
individual or a team? Apart from someone with expertise in
evaluation processes and facilitation, do we also need someone
external with subject expertise in our area of work?
5. Formulating a written evaluation plan outlining the process: What are
the steps in such an evaluation process?
(for example:
⇒ revisiting our objectives,
⇒ establishing indicators for measuring achievements,
⇒ designing methods of measuring,
⇒ collecting and organising and presenting data,
⇒ assessing what has been achieved against what was intended,
⇒ analysing the difference between the two,
⇒ making recommendations)
What are the time frames of the evaluation?
6. Revisiting the project objectives (what was planned): Is our original
plan available? Do we have differing views on what was planned? Can
we revisit and agree now on what was intended (project outputs, use
and benefit)?
7. Redefining (if necessary) indicators to measure these objectives:
How can achievement of these objectives be measured? Can we

BAOBAB: Handbook on Implementation Management – 6. Evaluation 14
measure quantities and quality? Do we need direct or indirect
indicators of achievement? Do we need to survey perceptions and
opinions, as well as statistics and percentages?
8. Devising the methods of data collection: Questionnaires, interviews,
workshops, surveys? Does the chosen method give us the specific
information we want? Does the chosen method limit our responses?
How much does each method cost (in staff time, in consultants’ time,
cost of paper, processing of results)? Do we need extra expertise (in
organising and analysing data using computers for example)?
9. Preparing and testing the data collection methods chosen: Can the
respondees answer easily? Are the questions clear? Does the method
yield exactly the answers we want? What about language barriers?
Are cultural barriers involved? Is any training required by our staff
/ volunteers in order to use this method? If so what? (include any
training necessary at this stage)
10. Collecting data according to method(s) decided upon: Is our
collection method yielding the data we require? If not, can we make
amendments?
11. Analysing the data: Are our results different to those we anticipated
when we planned the project? If so, by how much? Can we explain
this? If so, how?
12. Preparing results for presentation (usually written report plus verbal
presentation): Who will read the report? Who is our audience? What
points need emphasis?
13. Making recommendations based on the results (include in report and
presentation): What needs to be changed? Do we need to re-plan? Do
we need more resources? Do we need to scale down our activities? Do
we need more or less staff? Do we need to close down a section? Do
we need to offer different services? Do we need to change our
service times? Do we need to market ourselves differently?

BAOBAB: Handbook on Implementation Management – 6. Evaluation 15
14. Reporting-back formally (present report to authorities with
summary, conducting workshop report back(s): Who needs to know
the results of the evaluation? Which decision-makers (politicians,
staff of other departments, government ministers, donors)?
15. Following-up on the decision-making based on the evaluation results:
What can change immediately without staff or cost implications?
What can be changed in the short-, -medium- or long-term? What
process(es) of change is required (resistance, support)? Should any
of the recommendations be dropped and for what reasons?

BAOBAB: Handbook on Implementation Management – 6. Evaluation 16
6.4 A piece of management “software”
Evaluation results are contained in a report: this requires communicating the
information / findings back to those who will use it in decision-making. This
is just as important with own internal evaluations as external. Here are some
hints on report writing.
Report Writing
Overview of problems, mistakes, and possible improvements
in report writing
1. Reader - Writer - Relationship
Difficulties Unconducive practices Recommendations
The message is conveyed
to (an unknown or at best)
a partially known reader
who is remote from the
writer in time and place
There is no intermediate
and/or immediate feedback
of the reader(s) to
the messages while they
are delivered
The reader(s) are left to
interpret the messages (=
duplicate and understand
them) by themselves
without the opportunity to
Not being aware of
the reader while
conceptualising a
report, while actually
writing, and while
revising an already
written draft
Not being aware of
readers’ reactions
Not being aware of
readers’ questions
Investigate and imagine
clearly who your possible
readers are and what
their interest in your
text is (e.g. do they have
to take a decision on the
basis of your
statements? do they
need comprehensive
information? do they
want an executive
summary?)
Pick the reader up where
you as the writer suppose
her/him to be; have a
‘mock-reader’ give you
feed-back
Write as if you dialogue
with the reader; put
yourself into his/her
shoes and ask yourself:
what would be her/his

BAOBAB: Handbook on Implementation Management – 6. Evaluation 17
seek clarification with the
author
2. Writing as a special type of communication
main question? which
questions would follow
after each of your
statements? (re-edit!)
Difficulties Unconducive Practices Recommendations
The statements are
unchangeable / cast in
letters by which the
author puts him-/herself
on the spot (positively as
well as negatively)
To be restricted to the
means of a series of
words only (without the
non-verbal parts of
messages like tone of the
voice, gestures, etc.)
The message usually
consists of a more
comprehensive amount of
statements, than verbal
messages do - or even of a
series of arguments / a
whole article or a book
Pre-emptive excuses for
clear statements so that
formulations are vague or
unduly rigid / general:
passive voice, impersonal
pronouns, nouns created
from verbs, safety-firstformulations
Not to write close to how
one talks but to try to
imitate ‘grand elegance’:
long sentences, difficult
words
Unclear writing reflects
unclear thinking (which is
not structured
sufficiently clearly
enough for the type of
writing in which it is
found)
(Style I) Use active
voice, personal
pronouns, action
verbs,
(Style II) Make it
short and sweet: write
short sentences, only
put the minimum
necessary information
into a sentence / a
paragraph / your
whole text
(Style III) Think
before / while writing:
explain the
background / purpose
of your message; give
overviews; make
conclusive summaries;
structure the
sequence of your
arguments
convincingly

BAOBAB: Handbook on Implementation Management – 6. Evaluation 18
3. Prescribed formats for reports
Difficulties Unconducive Practices Recommendation
Formats for reports
are prescribed to
different degrees of
detail in order to
assure comparability
and/or quality of
contributions
6.5 Closure of the chapter
Mechanically follow
the headings given or
fill out the given
boxes
In this chapter you can expect to have learned:
♦ what evaluation is
♦ what different types of evaluation are practised
♦ when these are useful
Put your own messages
into the report format
while structuring them
according to the
prescriptions
♦ how evaluation relates to implementation and how it differs
from monitoring
♦ who evaluations are for
♦ who evaluates
♦ what the main / useful questions are in internal and external
evaluation
about some pointers that make for effective report writing.