Question and Questionnaire Design - Stanford University

Question and Questionnaire Design 297out by monitors as interviews are conducted or (more reliably) from recordings of theinterviews. The most basic code (e.g., Fowler & Cannell, 1996) identifies departuresthe interviewer makes from the question wording as well as departures therespondent makes from a satisfactory answer, for instance, requesting clarificationor expressing uncertainty. 23 Hess, Singer, and Bushery (1999) found that problematicrespondent behavior as measured by behavior codes was inversely related to anitem’s reliability. Dykema, Lepkowski, and Blixt (1997) found that severalrespondent behavior codes were associated with less-accurate answers (though, forone item, substantive changes in the interviewer’s reading of the question wereassociated with more accurate answers).Response latency measures the time it takes respondents to answer a question.It may be assessed either during an interview by the interviewer’s depressing a keywhen she finishes asking an item and then again when the respondent begins hisanswer, or after the interview is completed by listening to recordings (which, as withbehavior coding, is less error-prone). Unfortunately, the interpretation of longertimes is not always straightforward, as delays in responding could mean that aquestion is difficult to process (usually a bad sign) or that the question encouragesthoughtful responding (typically a good sign). The one study we know of thataddresses this issue with validation data (Draisma & Dijkstra, 2004) found thatlonger response latencies were associated with more incorrect answers, thoughanother study that addressed the issue more indirectly (Bassili & Scott, 1996)reported mixed results.Vignettes describe hypothetical situations that respondents are asked to judge.They have been adapted to pretesting to gauge how concepts conveyed in questionsare understood. In their test of the meaning of the Current Population Survey’s workitem, for example, Campanelli, Rothgeb, and Martin (1989) administered vignetteslike the following:I asked you a question about working last week. Now I’mgoing to read a list of examples. After each, please tell mewhether or not you think the person should be reported asworking last week. (1) Last week, Susan only did volunteerwork at a local hospital. Do you think she should be reported asworking last week? (2) Last week, Amy spent 20 hours at homedoing the accounting for her husband’s business. She did notreceive a paycheck. Do you think she should be reported asworking?23. More elaborate behavior codes (e.g., van der Zouwen & Smit, 2004) consider interaction sequences,e.g., the interviewer reads the question with a minor change, followed by the respondent’s request forclarification, which leads the interviewer to repeat the question verbatim, followed by the respondentanswering satisfactorily.