Technical Note: Conducting Mixed Method Evaluations - usaid

Technical Note on Conducting Mixed-Method Evaluations
(2) When different methods are used to answer the same elements of a single question, increasing confidence
in the validity and reliability of the evaluation results.
(3) When the results from one method are used to help design future phases of the evaluation using other
methods.
In addition to these three main reasons, there are other benefits that can be realized by using mixed-method
designs or data collection strategies. For example, mixed-methods approaches:
Are more likely to reveal unanticipated results.
Can provide a deeper understanding of why change is or is not occurring as planned.
Often capture a wider range of perspectives than might be captured by a single method.
(1) USING DIFFERENT METHODS TO ANSWER DIFFERENT QUESTIONS OR TO
ANSWER DIFFERENT PARTS OF THE SAME QUESTION
In many cases, one evaluation method will be insufficient to answer all of the questions included in an
evaluation statement of work (SOW). For example, suppose an SOW involves an evaluation of a project that
includes a new teaching technique, and includes two questions: #1. “Was there a statistically significant
difference between female and male students’ academic achievement test scores?” and #2. “How did students’
parents perceive the effects of the project?” A single method will likely be insufficient to adequately answer
both of these questions.
To answer the first question, the evaluator might choose a quasi-experimental design that uses existing test
scores from before project implementation and new test scores from after completion to compare the
performance of male and female students. This approach would address the question of whether the program
resulted in differences in test scores between females and males. But these methods would not help to answer
question #2. To understand parent perceptions, the evaluator likely would use individual or focus group
interviews of a sample of parents, and perhaps conduct an evaluative case study in order to more deeply
understand how parents view the program.
Sometimes, one evaluation question may contain multiple parts, and it
may be necessary to use different methods to address each part. For
example, question #1 could be changed slightly to ask, “Was there a
statistically significant difference between female and male students’
scores? And, what explains possible gender differences in test scores?” In
this case, simply relying on test scores in a quasi-experimental design
would be insufficient. To understand the mechanism behind differences in
female and male scores would require a different method, such as key
informant interviews with teachers or focus group discussions with
students.
(2) USING DIFFERENT METHODS TO ANSWER THE SAME
QUESTION: TRIANGULATION
Even if an evaluation question can be answered using only one method,
often it is preferable to combine multiple methods to answer the same
A NOTE ON
TERMINOLOGY
In the literature on evaluation,
“method” is sometimes used to
refer to a data collection
technique (interviews, surveys,
observations), and other times to
an evaluation design or approach
(experimental, quasi-experimental,
non-experimental). Though the
definition is not completely settled
in the literature, this Technical
Note treats evaluations that
combine methods in either sense
as mixed-method evaluations.
question in order to gain a more complete understanding of the issue and more confidence in the findings. By
approaching the same question from more than one perspective or by using more than one technique,
evaluators can then compare and contrast the results from these different methods. This process is known as
triangulation. If the findings from the different methods are similar, or reinforce one another, then users can
have greater confidence in the findings than if they are based on only one method.
2