Module 8: Data Collection Methods

Module 8: Data Collection Methods

IPDETModule 8:Data Collection MethodsInterventionorPolicyEvaluationQuestionsDesignApproachesCollectParticipatoryAvailable DataObservationSurveyQuestionnaireFocus GroupsDiaries, Journals, SelfreportedChecklistsExpert JudgmentDelphi TechniqueCitizen Report Cards

Introduction• Data Collection Strategies• Data Collection General Rules• Key Issues about Measures• Quantitative and Qualitative Data• Introduction to The Toolkit• Techniques for Using Surveys• Techniques for Using Focus GroupsIPDET 2 2

No One Best Way• Decision Depends On:– What you need to know• numbers or stories– Where the data reside• environment, files, people– Resources and time available– Complexity of the data to be collected– Frequency of data collectionIPDET 3 3

Structured Approach• All data collected in the same way• Important for multi-site and clusterevaluations so you can compare• Important when you need to makecomparisons with alternate interventionsIPDET 4 4

When to Use StructuredApproach:• You wish to be very precise• You are working with a large sample orpopulations• You are sure of what you wish to measure• You want to show your results numerically• You need to make comparisons acrossdifferent sites or interventionsIPDET 5 5

Semi-structured Approach• Systematic and follow generalprocedures but data are not collected inexactly the same way every time• More open and fluid• Does not follow a rigid script– may ask for more detail– people can tell what they want in their ownwayIPDET 6 6

When to Use SemistructuredApproach:• You are conducting exploratory work in a newdevelopment area• You are seeking understanding, themes,and/or issues• You want anecdotes, stories, or in-deptinformation• You are not sure of what you wish to measure• There is no need to qualifyIPDET 7 7

Data CollectionGeneral Rules• Use available data if they already exist• If using available data, be sure to findout how they:– collected the data– defined the variables– ensured accuracy of the data(continued on next slide)IPDET 8 8

Data CollectionGeneral Rules (cont.)• If you must collect original data:– establish procedures and follow them– maintain accurate records of definitionsand coding– pre-test, pre-test, pre-test– verify accuracy of coding, data inputIPDET 9 9

Key Issues aboutMeasures• Are your measures credible?• Are your measures valid?• Are you measuring what counts?• Are your measures reliable?• Are your measures precise?IPDET 10 10

Credibility• How trustworthy or believable is yourdata collection ?– Are the data you are collecting giving youinformation about the actual situation?• Make sure data you are collecting arerelevant and most importantinformationIPDET 11 11

Validity• Does the measurement actually measurewhat it is supposed to?• Two kinds of validity– face validity• contents of the test or procedure look like they aremeasuring what they are supposed to measure– content validity• content of the test or procedure adequately represents allthat is required for validityIPDET 12 12

Reliability• Term to describe the stability of yourmeasurement• Measures the same thing, same way inrepeated tests• Examples:– in sports, speed measured by stopwatch– birth weights of newborn infants– attendance rates at schoolsIPDET 13 13

Precision• How the language used in the datacollection matches the measure• Example:– if the question is about countries, themeasures must be at the national level– if the question is about people, themeasures must be on the individual levelIPDET 14 14

Quantitative Approach• More structured• Attempts to provide precise measures• Reliable• Harder to develop• Easier to analyzeIPDET 15 15

Qualitative Approach• Less structured• Easier to develop• Can provide “rich data” — detailed andwidely applicable• Is challenging to analyze• Is labor intensive to collect• Usually generates longer reportsIPDET 16 16

Which Data?If you:- want to do statistical analysis- know exactly what you want tomeasure- want to cover a large group- want anecdotes or in-depth information- are not sure what you want to measure Qualitative- do not need to quantifyThen Use:QuantitativeIPDET 17 17

Obtrusive vs. UnobtrusiveMethods• Obtrusive – observations are made ofbehavior with the participant’sknowledge– interviews, surveys, focus groups• Unobtrusive – observations done withoutthe knowledge of the participant– historical/document/archival data– watching participants at a distanceIPDET 18 18

How to Decide on DataCollection Approach• Choice depends on the situation• Each technique is more appropriate insome situations than others• Caution: All techniques are subject tobiasIPDET 19 19

Data Collection• Participatory Data Collection• Available Data• Observation• Surveys• Focus Groups• Diaries, Journals, Self-reported Checklists• Expert Judgment• Delphi Technique• Citizen Report CardsIPDET 21 21

Toolkit 1: ParticipatoryData CollectionData that are collected when interactingwith people• Examples:– transect walks– social mapping– community meetingsIPDET 22 22

Mapping• Collect and plot information on thedistribution, access, and use ofresources within a community• Useful tool to involve stakeholders– provides a way to work together– increases understanding of the community– generates discussions, verifies secondarysources of information, perceived changesIPDET 23 23

Mapping Process• Draws a picture of the community• Places resources and assets in their locations– individual assets: skills, talent, networks,money, etc.– civic assets: faith associations, clubs, socialgroups, etc.– institutional assets: businesses, schools,health services, public transportation, etc.– environmental assets: parks, roads, farmland,housing, clean air and water, etc.IPDET 24 24

Uses of Mapping• Global Positioning System (GPS)• Google EarthIPDET 25 25

Toolkit 2: Available Data• Example sources:– files/records– computer data bases– government reports– other reports or prior evaluations– census data– documents (budgets, organizational charts,policies and procedures, maps)IPDET 26 26

Using Agency Records• Agencies may have already collected thedata you need• May also have summaries and/or reports,such as:– internal management reports– budget documents– reports to the public or funding agencies• Key issues:– validity, reliability, accuracyIPDET 27 27

Advantage/Challenge:Available DataAdvantagesChallengesOften less expensive and fasterthan collecting the original datayourselfThere may be coding errors orother problems. Data may not beexactly what is needed. Youmay have difficulty gettingaccess. You have to verifyvalidity and reliability of dataIPDET 28 28

Toolkit 3: Observation• See what is happening– traffic patterns– land use patterns– layout of city and rural areas– quality of housing– condition of roads– conditions of buildings– who goes to a health clinicIPDET 29 29

Ways to ObserveUnobtrusiveParticipantObtrusiveNo one knows you are observingYou actually participate in theactivityThe people being observed knowyou are there to observe themIPDET 30 30

Guidelines for PlanningObservations• Develop a checklist to rate your observations• Develop a rating scheme• Have more than one observer, if feasible• Train observers so they observe the samethings• Pilot test the observation data collectioninstrument• For less formal approach, have a few keyquestions in mind when you arriveIPDET 31 31

Advantages andChallenges: ObservationAdvantagesChallengesCollects data on actual vs. selfreportedbehavior or perceptions. It isreal-time vs. retrospectiveObserver bias, potentially unreliable;interpretation and coding challenges;sampling can be a problem; can belabor intensive; low response ratesIPDET 32 32

Toolkit 4: Surveys• Great for asking people about:– perceptions, opinions, ideas• Less accurate for measuring behavior• Sample should be representative of thewhole• Big problem with response ratesIPDET 33 33

Methods for Surveys• In-person interviews• Mail /phone/Internet interviews orsurveys• Self-administered questionnairesIPDET 34 34

Structures for Surveys• Structured:– Precisely worded with a range of pre-determinedresponses that the respondent can select– Everyone is asked exactly the same questions inexactly the same way, given exactly the samechoices• Semi-structured– Asks the same general set of questions but mayleave many, it not all, of the answers open-endedIPDET 35 35

Structured vs.Semi-structured SurveysStructuredSemistructuredharder to developeasier to completeeasier to analyzemore efficient when working with large numbersa little easier to developlabor intensive to conductharder to analyze but provide a rich source of datasubject to bias in interpretingburdensome for people to complete as a selfadministratedquestionnaireIPDET 36 36

Advantages andChallenges of SurveysAdvantagesChallengesBest when you want to know whatpeople think, believe, or perceive,only they can tell you thatPeople may not accurately recall theirbehavior or may be reluctant to revealtheir behavior if it is illegal orstigmatized. What people think theydo or say they do is not always thesame as what they actually do.IPDET 37 37

In-person Interviews• Useful for in-depth understanding ofexperiences, opinions, or descriptions• Useful when other approaches do notwork• Should be conversational• Can be done individually or in groups• Can be structured or semi-structuredIPDET 38 38

Mail / Phone / InternetInterviews and Surveys• Literacy issues• Consider accessibility– reliability of postal service– turn-around time– do respondents have telephone access?– do they have Internet access?IPDET 39 39

Self-administeredQuestionnaires• Written surveys that the respondentcompletes• Can be structured, semi-structured, or acombination• Should be short (no more than 20 min.)• Usually include one or two open-endedquestions– help make respondents more comfortableIPDET 40 40

Advantages of Interviews• Can be structured, unstructured, or acombination• Can explore complex issues in depth• Forgiving of mistakes: unclear questionscan be clarified during the interview andchanged for subsequent interviews• Can provide evaluators with an intuitivesense of the situationIPDET 41 41

Challenges of Interviews• Can be expensive, labor intensive, andtime consuming• May not be able to explore why peoplehave different viewpoints• Selective hearing on the part of theinterviewer may miss information thatdoes not conform to pre-existing beliefs• Cultural sensitivity: gender issuesIPDET 42 42

Toolkit 5: Focus Groups• Type of qualitative research where smallhomogenous groups of people arebrought together to informally discussspecific topics under the guidance of amoderator• Purpose: to elicit reliable data, not justinteresting informationIPDET 43 43

Uses of Focus Groups• Help develop a survey questionnaire• Contextualize survey data• Be used in tandem with surveys• Be used as a separate data collectiontoolIPDET 44 44

Focus Group ProcessStepProcess1 Introduce the focus group meeting2 Have the participants introduce themselves3 Present the first question, it should be easy, an ice-breaker4 Ask the main questions5 Ask the last (summary) questions6 Ask if there are other comments or questions7 Write-up impressions, major issues and points ofdiscussionIPDET 45 45

Advantages and Challengesof Focus GroupsAdvantages Relatively quick and easy, may take less stafftime than in-depth, in-person interviews;provides flexibility to make changes inprocess and question; ability to exploredifferent perspectives; it can be funChallengesAnalysis is time consuming, participantsmight be different from rest of population; riskof bias in interpreting data; risk of group beinginfluenced by moderator or dominantmembersIPDET 46 46

Toolkit 6: Diaries, Journals,Self-Reported Checklists• Use when you want to captureinformation about events in people’sdaily lives• Participants capture experiences in realtimenot later in a questionnaire• Used to supplement other datacollectionIPDET 47 47

StepProcessGuidelines for Diaries orJournals1 Recruit people face-to-face• encourage participation, appeal to altruism, assureconfidentiality, provide incentive2 Provide a booklet to each participant• cover page with clear instructions, definitions, example• short memory-joggers, calendar3 Consider the time-period for collecting data• if too long, may become burdensome• if too short may miss the behavior or eventIPDET 48 48

Self-reported Checklists• Cross between a questionnaire and adiary• The evaluator specifies a list ofbehaviors or events and asks therespondents to complete the checklist• Done over a period of time to capturethe event or behavior• More quantitative approach than diaryIPDET 49 49

Advantages and Challengesof Diaries, Journals …Advantages Rich data can capture the details that mightbe quickly forgotten over timeGood for information on how people use timeHelps in collecting sensitive informationSupplements interviewsChallenges Requires literacyMay change behaviorData may be incomplete or inaccuratePoor handwriting, difficult to understandphrasesIPDET 50 50

Toolkit 7: Expert Judgment• Interviews with experts, one-on-one orin a panel• Can be structured or unstructured– everyone answers a set of specificquestions– or free flowing (focus group model)– or formal presentationsIPDET 51 51

Selecting Experts• Establish criteria for selecting expertsbased on:– recognized expert– areas of expertise– diverse perspectives– diverse political views– diverse technical expertiseIPDET 52 52

Advantages and Challengesof Expert JudgmentAdvantagesChallengesFast, relatively inexpensiveCommunication of final results cangive credibilityWeak for impact evaluationMay be based mostly on perceptionsWorth of data is only as good as theperceived credibility of the expertsIPDET 53 53

Toolkit 8: Delphi Technique• Enables experts who live in different locationsto engage in a dialogue• Experts asked specific questions• Answers are returned to a central source forthe evaluator to summarize and feed it backto the experts for further comments• No one knows who said what so conflict isavoided• Experts can agree or argue with others’commentsIPDET 54 54

Advantages and Challengesof Delphi TechniqueAdvantagesChallengesInexpensiveConducive to independent thinkingAllows sharing of informationJudgments of a selected group onlyTendency to eliminate extreme positionsTime consuming and requires skill incommunicationRequires adequate time and participantcommitmentIPDET 55 55

Toolkit 9: Citizen Report(Score) Cards• Used to collect citizen feedback on publicservices from actual users of a service• Can assess the performance of individualservice providers and/or compareperformance across providers• Can generate a database of feedback onservices that can then be placed in the publicdomainIPDET 56 56

Advantages and Challengesof Citizen Report CardsAdvantagesChallengesMixes focus groups andquestionnaire data collectionIncreases response ratesSimple communicationLocal conditions must be conduciveRequires a large sampleLack of predictability in how differentplayers respondIPDET 57 57

Developing the Survey• Define the purpose and objectives• Decide the important issues to be covered• Establish the relative weight of the differentmodules in the survey• Identify important issues within sectors• Question writers may need to learn moreabout how specific programs work• Once this background work is done, theactual writing of the survey may beginIPDET 58 58

Writing Questions• Progressively more detail is needed ateach level– Define overarching objectives– Balance between sectors– Balance within sectors– Write questions to study specific issues orprogramsIPDET 59 59

The Right People toDevelop Questions• Small group– will know what subjects are of interest(policy and analytic) to the evaluation– will have experience using data fromsimilar surveys on a variety of topics• Team must get input from policymakersand program managers• Use people outside to check for jargonIPDET 60 60

Field (Pilot) Testing• Test with small number of subjects,diverse areas and socioeconomicgroups• Field test should look at three levels:– as a whole– each section– individual questions• Make revisions based on results of testIPDET 61 61

Sequencing Questions• Get the respondents involved in the interviewas soon as possible• Before asking about controversial matters,first ask about some facts• Intersperse fact-based questions throughoutthe interview• Ask questions about the present beforequestions about the past or future• The last questions might be to allowrespondents to provide any other informationIPDET 62 62

General Guidelines forConducting Surveys (1 of 3)• Keep it simple, clear, easy, short• Locate other people who have done the kindof evaluation you are interested in and locatesurveys similar to what you think you want todo• Make sure people know why you are askingthem to participate• Ask questions that are easy to answer and donot frustrate respondent’s desire to be clear intheir responses(continued on next slide)IPDET 63 63

General Guidelines forConducting Surveys (2 of 3)• Do not ask respondents for information thatrequires them to go to a file or other source. Ifyou must do this, you need to let them knowin advance so the material can be assembledprior to administrating the survey.• Respect their privacy. Treat surveysconfidentially and have procedures in place toassure privacy. Make sure you can insureconfidentiality. Never promise anonymityunless it can be absolutely delivered.(continued on next slide)IPDET 64 64

General Guidelines forConducting Surveys (3 of 3)• Respect respondents’ time andintelligence• Tell them how they were selected and whytheir participation is important• Do no harm: keep responses anonymous.For example, in your report, use aggregateresponses; and assign an identificationnumber to the data and destroy the link tothe person’s nameIPDET 65 65

Interviewing• Use trained data collectors• Use as few data collectors as possible• Establish a protocol for data collection• Do a “walk through” or trial run– 3-4 face-to-face interviews– 10 questionnairesIPDET 66 66

Developing an InterviewStepProcedure1 Define the purpose of the interview. Link your purposeto the evaluation objectives2 Decide on open-ended or close-ended questions3 Draft interview questions and sequence the questions4 Prepare an Introduction and Closure for the interview5 Prepare to record responses6 Pre-test the instrumentIPDET 67 67

Conducting Interviews• Let the interviewees know:– why they are being interviewed– how they were selected– how the data will be used– whether it is confidential– how long the interview will take– whether you might want to talk to themagainIPDET 68 68

Conducting Interviews:Set Up• Try to pick a time a place that is quiet andfree of distractions• Ideally, have a second person to help takenotes• Tape recording might be a possibility; makesure you check with interviewee and getpermission before recording– might want to ask when setting up the appointmentIPDET 69 69

Interviewing Skills• Stick to your script– if asking close-ended questions, askexactly the way written– have a script for clarifications– if asking open-ended questions, go with theflow rather than directing it• Be aware of cultural norms: eye contact,direct questions, gender issuesIPDET 70 70

More Interviewing Skills• Balance: if you ask about what theything are the major supports, follow withwhat you think are the major barriers• Try to avoid asking “why” questions,some may find this aggressive or critical• Accept whatever they say with empathyand without judgmentIPDET 71 71

Conducting Interviews:Added Touches• Share interview questions ahead of time– no surprises• Offer to share a summary of what youunderstand from the interview– this might be especially useful to give theinterviewee (especially if a high rankingofficial) a greater feeling of control• Thank you note afterwardsIPDET 72 72

Interviewing: TakingNotes• Take good notes without distracting from theconversation– maintain eye contact while writing– write key words or phrases, not verbatim– but — if someone is saying something you want tocapture, it is OK to ask them to repeat it or to finishwhat you are writing before asking the nextquestion– may want to ask “May I use your exact words?”IPDET 73 73

Writing-up the Interview• Every word and idea is valuable• Take time to write up your notes ascarefully and in-depth as possible• It is best to do at least a brief clean-upof notes immediately afterwards (leavean hour between interviews)• Write up full notes within a day of theinterview: memory decay sets in quicklyIPDET 74 74

Writing QuestionnaireSurveys• Must be understandable to everyone• If possible use an existing questionnaire as aguide• Make the survey easy to complete• Ask general questions first, thendemographic, then more specific, then a finalopen-ended question or two• Have draft questions reviewed by experts• Pre-test, pre-test, pre-testIPDET 75 75

Questionnaires Tips andTricks• Use simple, clear language, appropriate foraudience• Ask only one question at a time• Write your questions so that all feel theirresponses are acceptable• When possible, write questions so thatresponses range from negative to positive(continued)IPDET 76 76

Questionnaires Tips andTricks (cont.)• Avoid “yes” or “no” responses• Avoid absolutes at either end of thescale (always, never), consider usingscales• Ask questions about the currentsituation• Leave exits (no opinion)• Avoid using double negativesIPDET 77 77

Planning and Logistics• Need several weeks to plan• Focus group sessions usually 1 to 2 hours• Some up to 3-5 hours with planned break(executives, in-house management, highlevelparticipants)• Have clear starting and stopping times• Consider need for food and transportationIPDET 78 78

Facilities and Materials• Ideal – commercial facility designedspecifically for focus groups• Neutral, accessible, hotel meeting room,school or church meeting area• Table and chairs (comfortable setting)• Name tents• Laptop computer for taking notes (if possible)• Consider need for video or audio taperecordings or have additional people takenotesIPDET 79 79

Number of Focus GroupSessions• No fixed rule• Generally, do sessions until no newissues emerge or when commonthemes are consistent– usually 3 to 6 group sessionsIPDET 80 80

Ground Rules• “What is said here, stays here”• Everyone is encouraged to participatebut not everyone has to answer everyquestion• Respect different viewpoints• There are no right or wrong answers• One person speaks at a timeIPDET 81 81

Typical Questions• What did you learn at the teacherscollege that is most helpful to you inteaching primary school?• What are your greatest challenges inteaching primary school students?• What should the college teach so thatgraduates are prepared to meet thecurrent challenges?IPDET 82 82

Facilitator/Moderator Role• Be familiar with the script, rather than readingit, so the session appears conversational• Make sure everyone is heard• Manage time, closing off discussion, movingto next topic• Set ground rules• Say as little as possible• Keep personal views outside the room• Use active listening• Probe for elaboration (Tell me more)IPDET 83 83

Write-up• Write-up your impressions immediately aftereach focus group– major issues, major points of discussion– anything unusual that happened• Compare with partner• If recorded, make arrangements to have ittranscribed, or to carefully watch/listen to itlater• Identify themes and emerging issuesIPDET 84 84

Data Collection Summary• You can choose more than one datacollection technique, multiple tools oftenhelp you meet the evaluation needs• Do not let the tool drive your work• Choose the correct tool to meet theneeds of the evaluationIPDET 85 85

To continue on to theNext Module click hereTo return to theTable of Contents click here

More magazines by this user
Similar magazines