Questionnaire Design Clinic - Survey Research Laboratory

Introduction toQuestionnaire DesignSpring 2009 Seminar SeriesSurvey Research LaboratoryUniversity of Illinoiswww.srl.uic.edu1

Open vs. Closed Questions• Open-Ended Questions• Suggest no alternatives• Probing• Extensive interviewer training• Elaborate coding schemes• Expensive and time-consuming• Free exchange of ideas without biasSurvey Research Laboratory2

Open vs. Closed Questions• Closed-Ended Questions• Standardized questions and answers• Simple interviewer training• Easy to analyze• May make it too easy to answer without thinking• May limit spontaneity• Not best when• asking for frequency of sensitive behaviors• there are numerous possible responses• Role of initial question development and pretestingSurvey Research Laboratory3

Cognitive Steps in AnsweringQuestions1. Understand question.2. Search memory for information.3. Integrate information into summaryjudgment.4. Translate judgment onto responsealternatives.“Optimizing”Survey Research Laboratory4

Cognitive Steps• Understand question.• Search memory for information.• Integrate information into summaryjudgment.• Translate judgment onto responsealternatives.“Weak satisficing”Survey Research Laboratory5

Cognitive Steps• Understand question.• Search memory for information.• Integrate information into summaryjudgment.• Translate judgment onto responsealternatives.“Strong satisficing”Survey Research Laboratory6

Factors Influencing Satisficing• Respondent motivation• Respondent ability• Task difficultySurvey Research Laboratory7

Response Effects Explained bySatisficing• Response order• Acquiescence• Non-differentiation• Saying “don’t know”Survey Research Laboratory8

Response order effects• Primacy• RecencySurvey Research Laboratory9

Response Effects Explained bySatisficing• Response order• Acquiescence• Non-differentiation• Saying “don’t know”Survey Research Laboratory10

AcquiescenceForm A (agree I’s version):Individuals are more to blamethan social conditions forcrime and lawlessness in thiscountry.Form A (Agree I’s)Agree (I): 59.6%Disagree (SC): 40.4100(473)Form B (agree SCs version):Social conditions are more toblame than individuals forcrime and lawlessness inthis country.Form B (Agree SCs)Agree (SC): 56.8%Disagree (I): 43.2100(472)Survey Research Laboratory11

AcquiescenceAbortion should be legal.Strongly Agree Neither Disagree StronglyagreedisagreeDo you favor or oppose legalized abortion?FavorOpposeSurvey Research Laboratory12

Response Effects Explained bySatisficing• Response order• Acquiescence• Non-differentiation• Saying “don’t know”Survey Research Laboratory13

Non-differentiationHow important is each of the following qualities fora child to have?NotimportantExtremelyimportantGood manners 1 2 3 4 5 6 7Tries hard to succeed 1 2 3 4 5 6 7Is honest 1 2 3 4 5 6 7Is neat and clean 1 2 3 4 5 6 7Has sound judgment 1 2 3 4 5 6 7Survey Research Laboratory14

Response Effects Explained bySatisficing• Response order• Acquiescence• Non-differentiation• Saying “don’t know”Survey Research Laboratory15

Other Issues• Double-barreled questionsQuestion:Compared to one year ago,are you paying more, less, orabout the same for your autoand life insurance?Better Question:Compared to one year ago, areyou now paying more, less, orabout the same for…a. auto insurance?b. life insurance?Survey Research Laboratory16

Other Issues• Specify who, what, when, where andhowQuestion:What is your total householdincome?Better Question:In 2002, what was your totalhousehold income, before taxes?Please count income from allmembers of your household,including wages fromemployment, disability, socialsecurity, and public aidSurvey Research Laboratory17

Other Issues• Specify through cuesQuestion:During the past 30 days, didyou drink any beer?Better Question:People drink beer in many places– for example, at home, atrestaurants, at bars, sportingevents, at friends’ homes, etc.During the past 30 days, did youdrink any beer?Survey Research Laboratory18

Other Issues•Use words with singular meaningsAmbiguous:How would you comparehow close you are to familyin your current neighborhoodto how close you were inyour old neighborhood?Would you say your family iscloser here, further here, orthe same?Clearer:Compared to your lastneighborhood, do you nowlive closer to your family,are you further from yourfamily, or are you aboutthe same distance?Survey Research Laboratory19

Other Issues• Response Options• Should be consistent with the question• Should typically be exhaustive, includingevery possible answer• Categories should typically be mutuallyexclusive (no overlap)Survey Research Laboratory20

Other Issues• Constructing Response Scales• Length of response scale• Using graphic images, such asthermometers and ladders, and cardsorting for complex ratingsSurvey Research Laboratory21

Other Issues• Numbering vs. Labeling• Numbers alone• Partial labeling• Full labelingSurvey Research Laboratory22

Physical Format Checklist• Number all questions sequentially• Use large, clear type; don’t crowd• ‘White space:’ Place more blank spacebetween questions than betweensubcomponents of questions• List answer categories vertically instead ofhorizontally• Avoid double/triple ‘banking’ of responsechoicesSurvey Research Laboratory23

Physical Format Checklist (continued)• Be consistent with direction of responsecategories• Don’t split questions across pages• If necessary (e.g., question requires 1.5 pages), restatequestion and response categories on next page• Put special instructions on questionnaire asneeded, next to question• Distinguish directions from questions – beconsistentSurvey Research Laboratory24

Key References To Get StartedDillman, Don. Mail and Internet Surveys: The TailoredDesign Method. New York: John, Wiley & Sons, Inc,1999.Bradburn, N, Sudman, S. and Wansink, B. AskingQuestions: The Definitive Guide to Questionnairedesign – for Market Research, Political Polls, and Socialand Health Questionnaires. San Francisco: Jossey Bass,2004.See also past issues of Public Opinion QuarterlySurvey Research Laboratory25

Other ReferencesSchuman, H., & Presser, S. (1981). Questions and answersin attitude surveys. New York: Academic Press.Krosnick, J.A. (1991). Response strategies for coping withthe cognitive demands of attitude measures in surveys.Applied Cognitive Psychology, 5, 213-236.Krosnick, J.A., Holbrook, A. L., Berent, M. K., Carson, R.T., Hanemann, W. M., Kopp, R. J., Mitchell, R. C., et al.(2002). The impact of "No Opinion" response optionson data quality: Non-attitude reduction or an invitationto satisfice? Public Opinion Quarterly, 66, 371–403.Survey Research Laboratory26