Review Criteria for Peer Reviews of SOWs for USAID Forward ...

Review Criteria for Peer Reviews of SOWs for USAID Forward Evaluations
Evaluation Title: _______________________________________________
Evaluation SoW Review by: ______________________________________
Mission: ________________________
Date: ___________________________
1. Evaluation Purpose and Use COMMENTS
1.1. Does the SOW identify the evaluation as
either an impact or performance evaluation,
per the definitions in the USAID Evaluation
Policy?
1.2. Does the SOW identify the project or program
to be evaluated, and include all project
information (purpose, titles, project numbers,
dates, funding, implementing organizations,
and COTRs/AOTRs)?
1.3. Does the SOW state the purpose of the
evaluation?
1.4. Does the SOW indicate the audience for the
evaluation?
1.5. Recommended: Does the SOW discuss how
findings will be tracked and used?
2. Evaluation Questions
2.1. Does the SOW include a list of questions the
evaluation team is expected to answer?
2.2. Are the questions explicitly linked to specific
future decisions (e.g. how resources should
be allocated, how implementation should be
modified to improve effectiveness)?
2.3. In your opinion, are the key evaluation
questions limited in number, focused and
relevant to these future decisions? Experience
suggests limiting key questions to five or less.
2.4. Recommended: Does the SOW indicate
priority order of the evaluation questions?
3. Data Collection and Analysis
Methods
If Impact Evaluation:
3.1. Are the questions about measuring project
impact and attributing change to USAID
efforts?
3.2. Does the SOW either require defining a
counterfactual and establishing a control
group?
3.3. Does the SOW require specific experimental
or quasi-experimental methods or request
that prospective evaluators propose
experimental or quasi-experimental
methods?
3.4. In your opinion, are experimental or quasiexperimental
methods appropriate and
sufficient to answer the questions being
asked?
If Performance Evaluation:
3.5. Are the questions about project or program
performance, effectiveness, value, etc.?
3.6. Does the SOW specify qualitative and/or
quantitative data collection and analysis
methods or request that prospective
1

Review Criteria for Peer Reviews of SOWs for USAID Forward Evaluations
evaluators propose qualitative and/or
quantitative methods?
3.7. Recommended: Does the SOW request
methods in addition to document review, key
informant interviews, and site visits?
All Evaluations:
3.8. Does the SOW communicate methodological
strengths and limitations or request that the
prospective evaluators do so?
3.9. Does the SOW identify if baseline data is
available or how it will be collected if
necessary?
3.10. Does the SOW identify required data
disaggregation (e.g., by gender or geographic
region)?
3.11. Does the SOW describe other data analysis
requirements (e.g. purposeful or random
sample design, sample size, statistical power,
statistical significance, etc.)?
3.12. In your opinion, are the methods likely to
generate high quality and credible evidence
that correspond to the questions, taking into
account time, budget, and data availability?
3.13. Recommended: Does the SOW identify data
sources for each question and/or method
including existing administrative, nationallevel,
monitoring, or other data?
4. Evaluation Team Independence
and Qualifications
4.1. Is the evaluation team leader external
(defined as a third party contractor or grantee
managed directly by USAID)? If not, does the
SOW explain why (i.e., the evaluation is not
triggered by the evaluation policy, an external
evaluation not feasible, etc.) and what steps
will be taken to minimize bias and preserve
the independence of the team’s findings?
4.2. Does the SOW identify specific positions and
skills the team is expected to include e.g.,
evaluation expertise, technical, geographic,
cultural, language and other skills?
4.3. Does the SOW require team members provide
a written disclosure of conflicts of interest?
5. Stakeholder Participation and
Local Capacity
5.1. Does the SOW state that the written design of
the evaluation will be shared with countrylevel
stakeholders as well as with
implementing partners for comment before
being finalized per the Evaluation Policy? If
not, is there an unusual circumstance that
bars this?
5.2. Recommended: Is the SOW clear about
whether and how USAID expects its staff;
partners; customer/beneficiaries, local
evaluation experts, or other stakeholders to
participate?
6. Evaluation Timeline and Logistics
2

Review Criteria for Peer Reviews of SOWs for USAID Forward Evaluations
6.1. Is the evaluation timed to inform a key
project, program, or strategic questions and
decisions? Is there enough time to conduct
the evaluation prior to decisions being made?
6.2. Does the SOW specify start and completion
dates and total Level of Effort (LOE)?
6.3. Is the SOW clear about the program period to
be evaluated?
6.4. Is the total estimated budget consistent with
the LOE and methods specified in the SOW?
7. Evaluation Deliverables
7.1. Does the SOW include criteria for evaluation
reports in Appendix 1 of the Evaluation
Policy?
7.2. Does the SOW request a written design which
includes key questions, methods, main
features of data collection instruments, and
data analysis plan?
7.3. Does the SOW request an executive summary
3-5 pages in length that summarizes key
points (project purpose and background, key
evaluation questions, methods, findings, etc.)
7.4. Does the SOW require a final report with (at
minimum) the following annexes:
a) The Evaluation Scope of Work;
b) Any “statements of differences” regarding
significant unresolved difference of
opinion by funders, implementers, and/or
members of the evaluation team;
c) All tools such as questionnaires, checklists,
survey instruments, discussion guides; and
d) All sources of information, properly
identified and listed.
7.5. Does the SOW require all data and records
from the evaluation be provided to the COTR
(e.g., survey responses, interview transcripts)?
All quantitative data collected by the
evaluation will be: provided in an electronic
file in easily readable format; organized and
fully documented for use by those not fully
familiar with the project or the evaluation;
owned by USAID and made available to the
public barring rare exceptions.
7.6. Recommended: Does the SOW identify
deliverables in addition to a draft and final
version of the report? (e.g., detailed
evaluation work plan, summary of findings
prior to drafting the report; oral briefings for
stakeholders).
7.7. Recommended: Does the SOW request
creative deliverables such as webinars, pod
casts, videos, social media dissemination
strategy, summaries translated into local
language, etc.?
3