Evaluating Country Programmes - OECD Online Bookshop

Evaluating Country Programmes
80
This approach is seen in the IDB’s recent client surveys (IDB/Ecuador; IDB/
Dominican Republic). In each CPE, around one hundred “clients” (staff of the
partner country government financial institutions who worked on IDB-financed
projects) completed a long questionnaire. This asked them to categorise and prioritise
IDB aid instruments and to suggest possible alternatives.
An evaluation of EU aid to Ethiopia used pairwise ranking to identify aspects of
donor performance and matrix scoring to rate EU performance against other donors.
Groups of stakeholders (Ethiopian officials, donor staff, NGO and relief agency staff
and EU Delegation Advisors) were invited to compare pairs of donors and specify an
aspect of performance in which one performed better than the other. They then rated
EU country programme performance on each of these criteria from one (below
average for donors to Ethiopia) to three (above average) using matrix scoring.
Comparators and counter/factuals
Having established the conceptual nature of criteria by which country programme
strengths and weaknesses are to be judged and identified means to obtain
and present information on these performance criteria, it is then necessary to
establish a basis for comparison. Should the achievements of the country programme
be compared to other country programmes executed by the same donor;
to the country programmes of other donors in the same country (UNDP, 1997: 3); or
against a counter-factual (what would have occurred in the absence of the country
programme)? There are strengths and weaknesses to each approach.
If it is possible to find another donor with a similar programme operating in the
same country, this may be a major focus of the evaluation. This may be hard given
that country programme mix, priorities, target areas or target groups may differ considerably
between donors operating in any given country. Often, however, it is possible
to identify a suitable comparator for certain activities or sectors. This should
be attempted whenever possible. Few of those CPEs we reviewed attempted any
such comparison.
The counterfactual case (what would have happened without the country programme)
is extremely hard to determine. The World Bank notes that relating country
performance to Bank assistance is:
“a difficult topic, especially when the Bank’s influence on policy reform is
shared with the Fund and Government, when its lending role is overtaken by
international capital markets and other donors,…and when there are numerous
external shocks.”
(World Bank 1999: 4)
The temptation is to compare the country situation “before” and “after” the
period of donor assistance and use this to make broad conclusions about how the
situation “without” aid would have compared to that “with”. This is not a satisfactory
OECD 1999