Question and Questionnaire Design - Stanford University

294 Jon A. Krosnick and Stanley PresserSoviet Union should admit American reporters (Schuman & Presser, 1981). In thiscase, a consistency dynamic was evoked when the item came second (making acomparison explicit), but not when it came first (leaving the comparison implicitat best). 17In other cases, context can influence the meaning of response options by changingthe nature of the standard used to answer a question. For instance, ratings of BillClinton might differ depending on whether they immediately follow evaluations ofRichard Nixon or of Abraham Lincoln (cf. Carpenter & Blackwood, 1979).When question ordering affects the meaning of response options or the weightingof factors relevant to answering an item, one context does not necessarily yield bettermeasurement than another. Instead, the effects reflect the fact that choices — in ‘‘realworld’’ settings no less than in surveys — are often inextricably bound up with thecontexts within which the choices are made (Slovic, 1995). Thus, decisions about howto order items should be informed by survey aims. When possible, question contextshould be modeled on the context to which inference will be made. In an electionsurvey, for instance, it makes sense to ask about statewide races after nationwideraces, since that is the order in which the choices appear on the ballot. But in themany cases that have no single real-world analog, consideration should be given torandomizing question order. 18Although context effects can be unpredictable, they tend to occur almostexclusively among items on the same or closely related topics (Tourangeau, Singer, &Presser, 2003). Likewise the effects are almost always confined to contiguousitems (Smith, 1988; but for an exception to this rule, see Schuman, Kalton, &Ludwig, 1983). 19 Schwarz and Bless (1992) and Tourangeau, Rips, and Rasinski(2000) provide good theoretical discussions of survey context. An important toolfor identifying potential order effects in a questionnaire is pretesting, to which weturn next.9.10. Testing and Evaluating QuestionnairesNo matter how closely a questionnaire follows recommendations based on bestpractices, it is likely to benefit from pretesting: a formal evaluation carried out beforethe main survey. This is because best practice recommendations provide little17. Lorenz, Saltiel, and Hoyt (1995) found similar results for two pairs of items, one member of whichasked about the respondents’ behavior toward their spouse and the other of which asked about theirspouse’s behavior toward them.18. When the survey goal includes comparison to results from another survey, replicating that survey’squestionnaire context is desirable.19. With paper and pencil self-administration and some computerized self-administration, respondentshave an opportunity to review later questions before answering earlier ones. Thus, in these modes, lateritems can affect responses to earlier ones (Schwarz & Hippler, 1995), although such effects are probablynot common.