Learning from Nine Examples of Peacebuilding Evaluation

manage the process of contracts from start to end. The evaluation also found
that the terms “employed” and “unemployed” are often not meaningful in
Afghanistan. Instead, there is much more of a continuum as workers are increasingly
asked to work more as demand increases.
Considerations on the Methodology
Strengths
• The Evidence Summit was not designed to produce a rigorous peer review
of the evaluations, but in general, the survey appears to have been well
implemented. The data was checked for inconsistencies and problems,
and follow-up interviews were conducted with businesses when problems
emerged. This is good practice, although interestingly, Building Markets reported
that the managers often became suspicious or annoyed when the
return visit was conducted.
• The nature of the evaluation created strengths and weaknesses that are mirror
images. The evaluation was quite focused, seeking to illuminate a single
relationship, namely the link between winning a contract and producing jobs.
The strength of this approach in this case was two-fold. First, the relationship
examined was part of a well-specified theory of change. Second, the additional
linkages in that theory, for instance, from jobs to stability, are not only
plausible but also well supported by other research. 8
Challenges and Pitfalls
35
• The focused nature of the evaluation also limited what could be learned. The
evaluation, for instance, was not able to draw firm conclusions on the impact
of Building Markets’ activities themselves. There was no way to identify
through the survey what role the business portal or procurement training, for
instance, played in the businesses’ ability to win contracts. Building Markets
did a good job of acknowledging these limitations in the evaluation report
and did not claim too much or overstate results.
• The lack of control or comparison groups was noted. Interviewing businesses
that did not win a contract either through purposive sampling or a randomization
strategy would have improved Building Markets’ ability to answer a
broader range of questions, including the impact of their own programs.
• Commenters also raised the issue that the evaluation was conducted internally
and whether that may have biased results. Building Markets reported
that while the evaluation was conducted internally, it was also peer reviewed
8 See World Bank, The World Development Report 2011: Conflict, Security, and Development, http://wdr2011.
worldbank.org/fulltext. The report argues throughout that providing jobs is a key component of post-conflict
stability.