Evaluating Country Programmes - OECD Online Bookshop

OECD 1999
Country Programme Evaluation: A State of the Art Review
It is necessary to examine each agency’s own definition of the terms as there
is in most cases room for debate about whether a given item should be
regarded as (for example) an immediate objective or an overarching goal.
The Netherlands, for example, does not mention impact: however, in defining
effectiveness as “contribution to [the country’s] macro-economic self-reliance,
poverty alleviation, and the sustainability of project results”
(Netherlands/India, 1994: 50) it seems that it is talking not only about effectiveness
but also impact (and, in this case, sustainability too).
ii) There are problems in using logframe analysis, designed primarily as a tool
for project planning, to evaluate a complex country programme. A country
programme includes numerous, often imperfectly articulated or even contradictory
goals, to be achieved through a range of instruments, sectors and
channels. Few donors have attempted to apply criteria of efficiency, effectiveness
or impact to programme aid (White, 1996); technical assistance is
also hard to evaluate in logframe terms. This is a problem when attempting
to assess the relative contribution of programme aid and project aid components
of a given country programme (the problem of comparing “apples and
pears”: Maxwell, 1996), and when a country programme is predominantly
implemented as programme aid.
Thus, while there may be value in drawing on logframe-based concepts of the
relationship between criteria in country programme evaluation, there is also a need
to define all criteria in an explicit, logical and consistent way that is amenable to
measurement (or estimation). The intrinsic “evaluability” of activities may vary considerably
(IDB, 1998: 11-12).
In an effort to introduce consistency, DAC has provided a list of standardised
definitions of evaluation criteria (Box 2.3).
On the basis of these standardised definitions, we found the following criteria
presented in the descriptions of methodology in the CPEs we reviewed. It should
be noted, however, that in many cases a criterion was mentioned and defined but
not subsequently used to any significant extent in the analysis of the country programme
which followed.
Some studies also used other criteria based upon the particular circumstances
of the partner country. In evaluating aid given via multilaterals to Cambodia, a SIDA
study (not a CPE) addressed “additionality”: as many donors initiated aid to Cambodia,
SIDA wished to ensure that it gave aid to projects that would not have found
another donor (SIDA/Cambodia, 1995). This could arguably be seen to be a subset
of the criterion of “relevance”.
From project to country programme
Country programme evaluations vary in the degree to which they attempt to
provide an evaluation of the country programme which is “more than the sum of the
71