¿MISION CUMPLIDA? EVALUACIÓN DEL PROGRAMA DE ... - Novib

¿Misión cumplida? Evaluación Programa ON-LAC
involved actors, their policies and the wider environment. This allows for greater
flexibility in the evaluation judgements regarding the three Aims, which may well result in
higher degrees of effectiveness in some areas than in others. This holds in particular for
the step of aggregating results from projects to Programme, since the objectives of
individual projects show various degrees of adding up. In some cases, such as
international campaigns, aggregating the effects of single interventions is simply not
meaningful.
The criterion of efficiency has been applied from different angles. Programme and project
resources can be assessed in regard to the size of either target groups or employment
offered by counterpart organisations. In a more or less standardised sector (such as
health, education or microfinance), the deployment of resources is routinely
benchmarked, monitored and ex post accounted for. In more diffuse environments
encompassing various sectors and, as in the case of Aim 1 (Food security and income
generation) a host of intervention areas, standards are much harder to define and apply
for performance measurement. Complementary means in this evaluation were applied to
verify efficiency levels, such as checks on whether efficiency enhancing measures were
built into projects. A check on coordination efforts between counterparts is also helpful to
ascertain that resources have been well spent.
In the evaluation, the impact criterion has been pragmatically dealt with, using the
mainstream distinctions of primary vs. secondary, intended and unintended, direct and
indirect effects. Apart from semi-structured interviews with selected individuals and
meetings with focus groups, the work did not embark on primary data collection from
target groups. Secondary information sources rarely included impact evaluations (only
two cases), while none of these measured “differences in differences”, i.e. measuring the
effect of interventions at different points in time with counterfactual evidence from
control groups. Even if the results of such impact measurement had been available, the
robustness of the results – and their scope for successfully repeating practices elsewhere –
would have led to serious misgivings. Impact assessments, due to the lack of contrafactual
evidence, are therefore mainly based on the criterion of plausibility, since the lack of hard
evidence is compensated by considering factors which may help to infer that the
Programme has reasonably produced the intended outcomes.
The sustainability of the ON-LAC Programme after completion is traditionally evaluated by
circumscribing the benefits of intervention exclusively within the area of tangible project
results. In most of the analyses applied in this evaluation under the three Aims, the
concept refers to project outcomes as well as the institutional setting in which
counterparts have been acting. The inclusion of counterparts’ institutional strengths is
directly related to the relevance of the Programme, as numerous objectives in the future
will only remain achievable when worked on by capable and well-equipped organisations.
The issue of sustainability also depends on the reversibility of outcomes. The benefits of
interventions are unsure when counterpart organisations face the prospect of having to
reduce their scope of operations.
The former point in particular relates to gender mainstreaming, which as a cross-cutting
theme in this evaluation was incorporated into both project and counterpart analysis.
Even though relevant information in this respect was incomplete and not always
comparable, the changes in gender equilibrium were assessed in the area of employment,
conditions and influence, both within counterpart organisations and the universe of target
groups.
19