Evaluations of NGO International Development and ... - Bond

Evaluations of NGO International Development and HumanitarianWorkChristina Laybourn for BondThis briefing paper offers a concise overview of current practices and debates in the field ofNGO evaluations, particularly of development and humanitarian work. Whilst it is by nomeans comprehensive, it aims to present pertinent issues and stimulate discussion. The firstsection provides an overview of evaluation mechanisms, methodologies, and qualityinitiatives. The second section summarises some of the most common challengesexperienced by NGOs conducting evaluations, drawing on comments from NGOprofessionals, websites and research articles. Finally, there is a short bibliography of existingtoolkits and standards, and relevant reading, where the issues can be explored in moredetail.Evaluation PracticeEvaluation Mechanisms:As monitoring and evaluation has become increasingly used by NGOs around the world,many mechanisms have been developed to facilitate the process. Some of the most commonapproaches include:Performance indicatorsThe logical framework approachTheory‐based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysCost‐benefit and cost‐effectiveness analysis Impact evaluation 11 More details, and the advantages and disadvantages of each approach can be found here: Monitoringand Evaluation: Some tools, methods and approaches, The World Bankhttp://lnweb18.worldbank.org/oed/oeddoclib.nsf/24cc3bb1f94ae11c85256808006a0046/a5efbb5d776b67d285256b1e0079c9a3/$FILE/MandE_tools_methods_approaches.pdf1

There are now a wide‐range of training workshops and toolkits to help NGOs implementthese M&E strategies, however, evidence suggests that implementing an effective M&Esystem takes considerable time and support 2 . As will be discussed later, there are stillconsiderable challenges to NGO evaluations, however some NGOs have developed internalevaluation standards to ensure that their assessments are of a high quality. For example,Oxfam GB has developed Programme Evaluation Standards which outline Oxfam’s policy onhow to conduct programme evaluations, and how to respond to and share evaluationfindings 3 .Evaluation MethodologiesSelf‐assessmentAs confidence and training in M&E methods grows, NGOs are increasingly choosing toconduct evaluations of their work themselves 4 : asking staff to assess programmes orprojects, and submit reports to senior management. Self‐assessment has the considerableadvantage of being cheaper to implement than hiring an independent consultant. It also canencourage learning, and a sense of ownership of the evaluation process amongst staff.Agency staff may also be better placed to understand the complexities of a project orprogramme than an external evaluator.However, NGOs can find that self‐assessments can be difficult to conduct effectively, andinternal evaluators may miss some things that an outside observer would pick up on. Theyalso lack a guarantee of impartiality. For these reasons self‐assessments can have lesscredibility.Peer‐assessment (also know as joint evaluation)Peer‐assessment involves staff from one or more peer organisation conducting theevaluation. This means that, again, costs are kept relatively low, however evaluations bypeople external to the organisation can be more insightful, and more credible. Because theassessors come from similar organisations, they will be well informed about the challengesof the sector.The principle difficulty with peer assessment is that it requires a high level of trust betweenNGOs, so that the assessed organisation can be open and honest during the evaluation.Where there is strong competition for funds, for example, NGOs may be unwilling to let a2 Jean Ellis, Accountability and Learning: developing monitoring and evaluation in the third sector, CES(2008) http://www.ces-vol.org.uk/index.cfm?format=4883 Oxfam GB Programme Evaluation Policy http://oxfam.intellidirect.com/e/d.dll?m=234&url=http://www.oxfam.org.uk/resources/evaluations/downloads/oxfam_evaluation_policy.pdf4 Monitoring and Evaluation in the Third Sector: Survey Findings Charities Evaluation Services (2007)p2 http://www.ces-vol.org.uk/index.cfm?format=1512

potential rival organisation scrutinise their operations, in case the information is later usedagainst them.Large NGOs sometimes opt for an approach halfway between the two above: sending stafffrom one country office to evaluate the work of another office. This combines both thebenefits of familiarity with the organisations systems, and the insight of an outsideperspective.Peer‐reviewA slightly different approach is where an NGO submits their self‐assessment to a peerorganisation for review, or where a peer conducts an assessment of an NGO’s evaluationsystem. As will be discussed later, this is an effective way of maintaining evaluationstandards, but does not actually involve the peer assessing the NGO’s work.External‐assessmentAn NGO may commission an independent consultant to conduct an assessment of some orall of its work. Although external‐assessment is increasingly giving way to self‐assessment, itis still often required by donors when large amounts of funding have been awarded.External‐assessments have more credibility, and, as with peer assessments, an outsideperspective can offer some useful insights and suggestions, that the organisation’s staff maynot have thought of.However, external assessments can be very costly: where the overall programme budget issmall to begin with, hiring an independent consultant may simply not be feasible.Quality AssuranceAs will be discussed later, one of the primary difficulties cited with NGO evaluations is theirvariable quality. Where evaluations lack reliable detail or analysis, their findings will not becredible or useful. Several quality assurance initiatives have been developed in an attempt toaddress this, although they are primarily intended for donor bodies, rather than NGOsthemselves.The OECD Development Assistance Committee (DAC) has produced a set of Principles forevaluating development assistance, intended for donor governments and multi‐lateralorganisations. First published in 1991, the Principles include impartiality, independence,credibility and usefulness. These Principles are now widely adopted by developmentagencies in DAC member countries, and by several multilateral/international financialinstitutions, such as UNDP, the World Bank, and the IMF. Members of the DAC conduct jointevaluations of peer members against the Principles. DAC has also recently published a set ofQuality Standards for Development Evaluation, which are intended to provide a guide togood evaluation practice. The Standards follow the structure of a typical evaluation process:defining purpose, planning, implementing, reporting, and learning from evaluation results.Although the DAC Principles and Standards are intended for DAC members, they can beadopted by other development partners. For example, the Humanitarian network ALNAP has3

adapted the DAC Principles for conducting evaluations in complex emergencies, into a guidefor Humanitarian agencies. DFID’s guidance notes for NGO evaluations also draw on the DACStandards.The UN Evaluation Group has also developed extensive guidance for conducting evaluations,including Standards, a Code of Conduct, Norms and Ethical Guidelines. All UN departmentsnow conduct evaluations according to these standards, meaning that evaluations fromdifferent departments can be collated and presented collectively to donors.Several donor agencies have systems in place to ensure the quality of their own evaluations,or those of their funded NGOs. For example, the Swedish development agency, Sida,commissioned an independent assessment of Sida evaluations in 2008, which assessed theircoverage, credibility and usefulness of the results information. The report found severalareas of weakness, and recommended improvements to Sida’s evaluation processes 5 .The Dutch Ministry of Foreign Affairs’ Evaluation Department (IOB) similarly conducted anassessment of the Ministry’s new Tailor‐made Monitoring system for NGOs, to assess itsbenefits and disadvantages 6 . The Dutch Ministry also asks NGOs in receipt of the largestgrants to submit their self‐evaluations to the IOB so that the quality of the evaluations canbe assessed.ALNAP has made notable efforts to monitor and improve the quality of evaluations byHumanitarian NGOs, beyond adapting the DAC Principles, as mentioned above. Between2001 and 2006 ALNAP conducted yearly “meta‐evaluations” of Evaluations of HumanitarianAid (EHA), using a Quality Pro Forma that they developed according to accepted evaluationgood practice. Details of the “meta‐evaluations” were published in ALNAP’s yearly Review ofHumanitarian Action. This autumn, ALNAP has begun a process of action research focusingon strengthening the evaluation capacities of humanitarian organisations, and has produceda draft paper and framework of how to improve the effectiveness and use of humanitarianevaluations 7 .5 Kim Forss, Evert Vedung, Stein-Erik Kruse, Agnes Mwaiselage, Anna Nilsdotter Are Sida EvaluationsGood Enough? An Assessment of 34 Evaluation Reports Sida (2008)http://sidapublications.citat.se/interface/frmoptimaker3.asp?doctype=3&order=createdateDESC&departmentid=298&topheight=55&headerheight=23&fotheight=0&leftframewidth=300&width=820&stylesheet=sida.css&frameout=0&language=14&login=True&username=sih6 ‘Maatgesneden monitoring, beperkte beleidsdoorlichting MFS’ (A Limited Policy review of MFS TailormadeMonitoring) (2009)http://www.minbuza.nl/nl/Organisatie/Evaluatie/Afgeronde_onderzoeken/2009/04/IOB_Maatgesneden_monitoring_beperkte_beleidsdoorlichting_MFS7 Alistair Hallam Strengthening organisational capacities for evaluation of humanitarian action (Draft)ALNAP (2010) http://www.alnap.org/pool/files/org-capacities-workshop.pdf4

Accountability and Learning ParadoxAnother of the common problems cited, is the sometimes contradictory purpose ofconducting an evaluation. Evaluations are intended to highlight both the successes andchallenges experienced during a programme of work. However, evaluations are also oftencommissioned at the request of donors. NGO professionals see this as creating dual pullbetween learning from mistakes, and accounting to your donor 14 . If future funding is in anyway influenced by the NGO’s previous achievements, there are clear incentives to emphasisesuccesses, and minimise failures. However, by not reporting or examining any challenges,valuable learning opportunities are missed, both by the subject NGO, and by others whoread the evaluation.Use of EvaluationsThe ALNAP report on the utilisation of evaluations, notes concern in the sector thatevaluation results will not be used 15 . Jonthanan Patrick, Evaluation Adviser at DFID,recognises this concern amongst NGOs that evaluations will “fall into a black hole” once theyare submitted to a donor agency 16 . The sense that submitting evaluations to donors ismerely a “check box” exercise, and that they are not properly read, may contribute to thevarying standards by which evaluations are conducted. Donor agencies such as DFID and theDutch Ministry of Foreign Affairs admit that they cannot yet adequately process and collateall the findings from evaluations that are submitted to them 17 . However, there are someefforts to collate findings from evaluations: the Norwegian development agency NORAD hascommissioned synthesis reports of evaluations of Humanitarian assistance, andEnvironmental Development assistance 18 .Learning from EvaluationsOne of the most concerning issues around evaluations of NGO work, is that the findings arenot shared, and recommendations are not implemented. Although evaluations are widelyperceived to function as a tool for learning, an ALNAP report found very little evidence thatevaluations are used by NGOs to bring about changes or improvements in operations 19 . Anevaluation of the quality of DFID’s evaluation reports and assurance systems similarly noted14 Summary Report: European M&E Workshop INTRAC 2005; p 8; Harry Derksen, ICCO, privateconversation 12/08/201015 Peta Sandison, ‘The Utilisation of Evaluations’ in ALNAP Review of Humanitarian Action 2005,ALNAP (2005) p9016 Jonathan Patrick, private conversation 26/10/1017 Jonathan Patrick, private conversation 26/10/10; Phil Compernolle, Evaluator, Policy and OperationsEvaluation Department, Dutch Ministry of Foreign Affairs, private conversation 03/09/1018 Humanitarian Response to Natural Disasters: A Synthesis of Evaluation Findings, NORAD (2009); ASynthesis of Evaluations of Environmental Development Assistance by Multilateral Organisations,NORAD (2009).19 ALNAP (2005)6