Module 6: Descriptive, Normative, and Impact Evaluation Designs

IPDETModule 6:Descriptive, Normative, andImpact Evaluation DesignsInterventionorPolicyEvaluationQuestionsDesignQuestionsElementsTypesKey Points

Introduction• What Is Evaluation Design?• Connecting Questions to Design• Design Elements• Types of Designs for Impact Evaluation• Key Points about DesignIPDET 2 2

Evaluation Design• The total process of specifying a planfor:– collecting data– analyzing data– reporting results– getting the results usedIPDET 3 3

Design ProcessQuestions Designs Methods Analysis ReportingIPDET 4 4

Approach to DevelopmentEvaluationFocus the EvaluationFocus the Evaluation▪ Purpose▪ Purpose▪ Terms of Reference▪ Terms of Reference▪ Program logic model▪ Program logic model▪ Program outcome model▪ Program outcome model▪ Specification of evaluation▪ Specification of evaluationquestionsquestions▪ Identification of stakeholders▪ Identification of stakeholdersDesign & MethodologyDesign & Methodology▪ Evaluation questions▪ Evaluation questions▪ Data collection design▪ Data collection design▪ Measurement strategy▪ Measurement strategy▪ Sampling strategy▪ Sampling strategy▪ Data Collection strategy▪ Data Collection strategy▪ Develop data collection instruments▪ Develop data collection instruments•Involve stakeholders•Involve stakeholdersUse EvaluationUse Evaluation▪ Communicate Findings▪ Communicate Findings▪ Feed-back▪ Feed-back▪ Decision-making▪ Decision-making▪ Action Plan▪ Action PlanReport FindingsReport Findings▪ Interpret the data▪ Interpret the data▪ Write report▪ Write report▪ Make recommendations▪ Make recommendationsGather & Analyze DataGather & Analyze Data▪ Gather data according▪ Gather data accordingto protocolsto protocols▪ Prepare data for▪ Prepare data foranalysisanalysis▪ Analyze data▪ Analyze dataIPDET 5 5

Design Matrix• Another organizing tool to help plan anevaluation• Organizes questions and the plans forcollecting information to answerquestionsIPDET 6 6

Matrix Elements• Design Matrix Planning Instrument for:• Major Issues Being Addressed• Major Assumptions Being Made• Questions• Sub-questions• Type of Question• Design• Measures or Indicators• Criteria for Normative QuestionsIPDET 7 7

Design Matrix Planning Instrument for:QuestionsSub-QuestionsType ofQuestionDesignMeasuresorIndicatorsCriteria forNormativeQuestionsIPDET 8 8

Matrix Elements (page 2)• Data Sources•Sample• Data Collection Instrument• Data Analysis• CommentsIPDET 9 9

DataSourcesSampleDataCollectionInstrumentDataAnalysisCommentsIPDET 10 10

Answering DescriptiveQuestions• Descriptive questions generally use nonexperimentaldesigns• Common designs for descriptive questions:– one-shot– cross-sectional– before-and-after– time series– longitudinal– case studiesIPDET 11 11

One-shot Designs• A look at a group receiving an intervention atone point in time, following the intervention• Use to answer questions such as:– How many women were trained?– How many participants received job counseling aswell as vocational training?– How did you like the training?– How did you find out about the training?IPDET 12 12

One-shot• Represented as:– X O 1• There is one group receiving thetreatment “X” and one observation “O”• There is no before treatment/intervention measureIPDET 13 13

Cross-sectional Designs• Also show a snapshot at one point in time• Also interested in sub-group responses• Often used with survey method• Subgroups may be:– age– gender– income– education– ethnicity– amount of intervention receivedIPDET 14 14

Cross-sectional• Evaluation question may focus on– participant satisfaction of services– why they did not use services– find out current status of people from anintervention a few years ago• Evaluation questions might be:– Do participants with different levels of educationhave different views on the value of training?– Did women receive different training services thantheir male counterparts?IPDET 15 15

Cross-sectional• Represented as:– X O 1O 2O 3• The observation is made after theintervention “X” and responses ofsubgroups (“O 1 , O 2 , O 3 ” and so on)receiving the interventions areexaminedIPDET 16 16

Before-and-after Designs• Also called pre- and post-designs• Ask about group characteristics• There is no comparison group• Evaluation questions:– Did program participants increase theirknowledge of parenting techniques?– What was the change in wages earned,two years after the training intervention?IPDET 17 17

Before-and-after Designs• Represented as:– O 1 X O 2• Observation, intervention, observationIPDET 18 18

Time Series Designs• Look for change over time• Purpose is to explore and describechanges over time – either after, orbefore and after the intervention• Can be used to discern trends• Often there are existing data that can beusedIPDET 19 19

Time Series• Evaluation questions:– What are the trends in child mortality ratesbefore and after and over time for anintervention?– What are the changes in participantattitudes over time towards womenentrepreneurs?IPDET 20 20

Time Series• Represented as:– O 1 O 2 O 3 X O 4 O 5 O 6• Several (three shown above)observations are made prior to theintervention and again three more timesafter the interventionIPDET 21 21

Longitudinal Study• A type of time series design• Repeated measures of the same variable aretaken from the study population• Panel design is one type of longitudinal studywhere a small group of people is tracked atmultiple points over time– almost always use qualitative questions (openendedsurvey questions, in-depth interviews, andobservation)– can give a more in-depth perspectiveIPDET 22 22

Longitudinal• Evaluation question:– How did the allocation of social benefitseffect families’ transition into and out ofpoverty?• a study looking at Poland’s family allowancefrom 1993 to 1996IPDET 23 23

Case Study Design• Descriptive case study• In-depth information is collected over time tobetter understand the particular case or cases• Useful for describing what implementation ofthe intervention looked like – and why thingshappened the way they did• May be used to examine program extremes,or a typical interventionIPDET 24 24

Case Study• Represented as:O 1O 2O 3IPDET 25 25

Answering NormativeQuestions• Similar to descriptive questions• Normative always assessed against acriterion:– a specified desired or mandatory goal,target, or standard to be reached• Generally the same designs work fornormative questions as descriptivequestionsIPDET 26 26

Answering Cause-EffectQuestions• Pose the greatest challenge• Need a well thought out design• Design attempts to rule out feasibleexplanations other than the intervention• Internal validity: a design’s ability to ruleout other explanationsIPDET 27 27

Common Threats toInternal Validity• History (events occurring at the same time)• Maturation of subjects (getting older changesthe results)• Testing (learning how to take the test)• Instrumentation (changes in data collectioninstruments or procedures)• Selection bias (participants may be differentto begin with)• Attrition (a specific group of people may dropout)IPDET 28 28

Impact Designs• Can use experimental and quasiexperimentaldesigns• Experimental sometimes called the“medical model”– randomly assign participants to a group,group does not know who is in thetreatment or placebo group (“blind studies”)IPDET 29 29

Controlling• To reduce the possibility of believing weknow something as true which is reallynot, need to control everything but theintervention, including:– the implementation of an intervention– who receives it– the environment in which it is deliveredIPDET 30 30

Impact and Multi-site andCluster Evaluations• Each site and the nature of theinterventions may vary in differentlocations• Complexity may limit options for designIPDET 31 31

Design Elements forImpact Questions• For evaluators doing traditional experimentalevaluation:– before-and-after measures– comparison groups– random assignment to the comparison groups• For newer approaches (i.e. cluster, multi-site,and rapid assessment)– use of control variables– use of natural variation– causal tracing strategiesIPDET 32 32

Before-and-AfterMeasures• Change is measured by comparing keymeasures after the intervention began againstthe measures taken before the interventionbegan• Before measure might be called the baseline• Collecting baseline data might be called abaseline study• Change alone does not prove causalityIPDET 33 33

Comparison Groups• Treatment group: group that receivedtreatment• Control group: group that does not receivetreatment• If the intervention causes change those intreatment group show more change than thecontrol group• Again, alternative explanations must be ruledout before drawing conclusionsIPDET 34 34

Random Assignment• Random: people, or things are placed ingroups by chance• Random assignment makes groupscomparable• Not always an option but it is possiblemore often than you think– when not all participants can receive theintervention at onceIPDET 35 35

Use of Control Variables• Random assignment impossible?– Rule out alternative explanations bystatistically controlling for them:• prior performance or prevalence levels• socioeconomic status• prior soil quality• weather / climateIPDET 36 36

Use of Natural Variation• Inconsistent implementation? Turn itinto an advantage• Useful evidence includes:– less extensive implementation• smaller (or no) impact– better quality implementation• more positive results and/or fewer negativeimpactsIPDET 37 37

Causal Tracing Strategies• Based on the general principles used intraditional experimental and quasiexperimentaldesigns, but:– can be used for rapid assessments– can be used without high-level statistical expertise– can be used on small scale interventions wherenumbers preclude statistical analysis– can be used for evaluations with a qualitativecomponent– involves the evaluator doing some detective workIPDET 38 38

9 Causal TracingEvidence Sources• Causal list inference• Modus operandi• Temporalprecedence• Constantconjunction• Contiguity ofinfluence• Strength ofassociation• Biological gradient• Coherence• AnalogyIPDET 40 40

Types• Experimental design• Quasi-experimental design• Correlational design• Case study design• Non-experimental designIPDET 41 41

Experimental Design• Called the “true experiment”– involves random assignment– uses comparison groups– often includes before-and-after measures• Considered the optimum approach butcan be difficult to implement• Drawback:– often small scale, less generalizableIPDET 42 42

Randomized ControlTrials (RCTs)• Consider when:– you have a discrete, concrete intervention –singular, well-defined– implementation can be standardized– valid and reliable measures exist for theoutcome to be tested– random assignment is possible– random assignment is ethicalIPDET 43 43

Randomized ControlTrials (RCTs)• NOT appropriate for:– complex, multi-dimensional and highly contextspecificcommunity interventions– ethical constraints• NOT needed if:– face validity is high– observed changes are dramatic– link between treatment and outcome is directIPDET 44 44

Quasi-experimentalDesign• The design is similar to trueexperimental design but:– no random assignment– uses naturally-occurring comparisongroups– requires more data to rule out alternativeexplanationsIPDET 45 45

Examples of QuasiexperimentalDesign• Before-and-after: good for descriptivequestions• Matched and non-equivalent comparisondesign• Time series and interrupted time seriesdesign• Correlational design using statistical controls• Longitudinal design• Panel designIPDET 46 46

Correlelational Design• Often used when seeking to answerquestions about relationships andassociations• Often used with already available dataIPDET 47 47

Case Study Design• Used when the researcher wants to gain anin-depth understanding of a process, event,or situation• Good to learn how something works or whysomething happens• Are often more practical than a national study• Can consist of a single case or multiple cases• Can use qualitative or quantitative methods tocollect dataIPDET 48 48

Key Points about Design• There is no perfect design• Each design has strengths and weaknesses• There are always trade-offs – time, costs,practicality• Acknowledge trade-offs and potentialweaknesses• Provide some assessment of their likelyimpact on your results and conclusionsIPDET 49 49

To continue on to theNext Module click hereTo return to theTable of Contents click here