Evaluating Country Programmes - OECD Online Bookshop

OECD 1999
Country Assistance Evaluation in the Multilateral Development Banks
Box 5.2. Measuring the counterfactual
Attempts to measure the counterfactual in the context of country assistance are still
in their infancy stage.No reliable methodology has yet been developed to measure
how a country would have evolved in the absence of development assistance from one
or more donors. The World Bank and others have used a number of different methods
in different studies to provide a counterfactual reference for a particular result.
1. Long-term growth models: Different economic long-term growth models
based on cross-sectional data have been used to forecast an economy’s
growth based on a limited number of economic resource, demographic and
technical assumptions. Examples include the Levine and Renelt equation and
the Barro equation which can be used to generate growth projections. These
models are based on long-term assumptions and do not take into consideration
factors such as external commodity prices, exogenous shocks, etc. which
can have short-term effects on an economy and change the pattern of growth.
2. Large-scale econometric models: In economics, “with or without” issues are
typically dealt with through the use of a multi-equation econometric model,
where simulations can be run using the two “with and without” cases. For some
developing countries, where an econometric model has already been developed,
different simulations could be run to determine how the economy
would have performed in the absence of the institution’s development assistance.
While the results are estimates based on a large number of assumptions
built into the model, they are typically more informative than “the second best
guess” of what would have happened.
3. Country comparisons: Country comparisons involve taking two or more countries
that are starting from the same point of development and comparing growth and
development outcomes and results. In some cases a country may have received
little or perhaps no assistance from an institution or external donors. This country
may then be compared to the country in question to provide a counterfactual reference.
A reference case could be Vietnam which received little external aid
through the late 1980s and 1990s but still experienced very high growth rates and
a wide distribution of benefits amongst its population.
4. Induced implicit assumptions: Another method of generating the counterfactual
reference is through implicit assumptions by the evaluator on why certain
operations were not successful. When it is clear that an operation did not
achieve the expected results, closer examination of the circumstance would
lead to an approximation of the probable state of the sector, region or country
in the absence of that operation.
5. Ex-ante counterfactuals in the preparation of country strategies and operations:
Ideally in the preparation of country strategies and operations, staff
would provide a scenario of how the country benefits from the proposed
project or programme or conversely where the country would be in the
absence of the proposed operation. These scenarios could be used to generate
the counterfactual reference of where the country would be in the absence
of institutional support.
149