Download PDF - ReliefWeb

population). A good practice case that provides general guidance but does not attempt to
micro-manage, providing flexibility for the evaluation team, is given in Box 2.7.
Team composition also followed the conventional pattern, with most teams made up of
consultants based in the country or continent of the commissioning agency. While there are
potential advantages to this arrangement, including the fact that evaluators may be a
‘known quantity’ to commissioning organisations, and that the pool of evaluation expertise
may be larger in the commissioning countries, there are also disadvantages. These include:
the higher costs associated with the use of ‘northern’ consultants; the limited time they are
able to spend on field work (often a result of fee rates and available resources); their
frequent lack of appropriate language skills; and the loss of opportunity to build up
indigenous capacity. Host country citizens, on the other hand, while not always having the
breadth of experience or evaluation background of ‘northern’ consultants, are likely to have
language skills and sound local knowledge and contacts – invariably key factors
contributing to an evaluation’s success.
As with technical assistance, the choice of team members for evaluations of humanitarian
action should complement and strengthen, rather than replace, indigenous capacity
(Sobhan, 1997). Of the 33 individual evaluation reports, 21 were carried out by expatriates,
11 by a mix of host-country citizens and expatriates, and only one by host-country citizens
(DEC, 2000), the latter specifically to increase host-country capacity. Mixed team reports,
i.e. those by host country and expatriate staff , did not appear to be significantly different in
quality to those carried out by expatriates only. This combination of team members may
currently be the most appropriate team make-up, given that it provides both international
and national expertise.
Only one-third of the reports note the expertise of team members. However, this generally
only relates to sectoral expertise which makes it difficult to come to any overall
conclusions about whether evaluation teams included relevant personnel and skills,
including language skills (see section on Legibility below).
Another feature in nine of the reports (three by NGOs, five by UN agencies, and one by a
bilateral donor) was the inclusion of agency staff on the evaluation team, though none
discussed the implications of this. While this may increase participation and the likelihood
of uptake of recommendations, it may also be seen as a method of biasing results and so
lower the credibility of the study. Reports ought to state explicitly the advantages and
disadvantages of including agency staff (particularly where the staff come from outside the
agency evaluation office) as ‘internal’ evaluative processes are often viewed as less
impartial than those carried out externally. As is well known, negative evaluation results
are subject to extensive discussion and negotiation, with evaluation offices often playing
the role of mediator between the evaluation team and the department or agency under
review. This process itself can lower the credibility of the evaluation because it may appear
to call into question the evaluators’ impartiality. It is therefore particularly important to
provide a clear rationale when including agency staff – particularly as inclusion of agency
staff is not a practice that is common in the general evaluation field 7 .