Evaluating Country Programmes - OECD Online Bookshop

OECD 1999
Country Programme Evaluation: Synthesis Report from the Workshop
flexibility in country programme management can make evaluation harder, yet this
should obviously not be used as an argument for decreasing the flexibility granted
to country programmes.
At the outset, then, it is recognised that assessing the impact of a country programme
is inevitably more complex, costly and time-consuming than other forms of
evaluation. However, for all the reasons outlined above, the effort is seen as justified.
If impact at the country level cannot be established scientifically or precisely
(i.e. cannot be measured), it can at least be estimated and described by postulating
and testing logical linkages between aid activities and observed changes. This
approach is apparent in, for example, the detailed breakdown of the effects of
World Bank advice and assistance to the Philippines (see Chapter 9).
Differing approaches to the identification and attribution of impact
Figure 1.2 provides a conceptual sketch that relates donor activities, nondonor
processes and change. The central message of this diagram is that aid results
in change in numerous and complex ways which may be hard to trace. It also draws
attention to the fact that aid is generally not the central influence upon change:
trends and shocks in the global economy, the capacity and will of the partner government
and other non-aid factors are of key importance in explaining change, with
aid generally influencing these processes only at the margin.
The ideal approach to impact assessment would thus be “top-down”, looking
at change in national conditions (economic performance, social development indicators,
etc.) and only then looking at what part(s) of this change could be attributed
to aid activities. This approach is described in Figure 1.2. as an inductive or historian’s
approach. It could in theory proceed without a statement of country programme
goals, i.e. bypassing the a priori cause-effect linkages required for logframe
analysis. It requires the construction of a counterfactual scenario – a hypothetical
model, either qualitative or quantitative – of what the national situation would have
been in the absence of aid.
Unfortunately it is extremely hard to construct a counterfactual case at the
country level. The IDB paper circulated at the Workshop summarised five
approaches to the modelling of a counterfactual in CPEs produced by the multilateral
development banks. These are presented in Box 1.3.
As listed in Box 1.3, a comparator provides a methodological short-cut to the
construction of a counterfactual. If two similar cases can be identified which demonstrated
similar initial conditions and trends and only one then received development
assistance, the impact of aid is “read off” – attributed – from the difference
between the case with and the case without. At a project level, it is possible but
hard to establish a counterfactual (e.g. by comparing change in a village which
receives assistance and one which does not). At a sectoral or country level, it is
25