Question and Questionnaire Design - Stanford University

300 Jon A. Krosnick and Stanley Presserquestions are likely to have been crafted by skillful artisans or those with manyresources to develop and test items.Moreover, even when questions from prior surveys depart from best practice, theymay be useful to borrow. This is because replicating questions opens up significantanalytical possibilities: Comparisons with the results from other times and fromother populations. As such comparisons require constant wording, it will beappropriate to ask questions that depart from best practice in these cases.Will such comparisons be affected by the response errors that arise from thedeparture from best practice? Not if the response errors are constant across thesurveys. Unfortunately, most of the literature on question wording and contextfocuses on univariate effects, so we know less about the extent to which responseeffects vary between groups (i.e., the effect on bivariate or multivariate relationships).Although there is evidence that some response effects (e.g., acquiescence) may affectcomparisons between certain groups (e.g., those that differ in educationalattainment), there is evidence in other cases for the assumption of ‘‘form-resistantcorrelations’’ (Schuman & Presser, 1981).Relevant evidence can be generated by repeating the earlier survey’s item on onlya random subsample of the new survey, and administering an improved version tothe remaining sample. This will not yield definitive evidence (because it relies on theuntested assumption that the effect of wording is — or would have been — the samein the different surveys), but it can provide valuable information about the measures.Second, just as different versions of the ‘‘same’’ item administered to split samplescan be instructive, multiple indicators of a single construct (administered to the entiresample) can likewise be valuable. Although the emphasis in the question literature isgenerally on single items, there is usually no one best way to measure a construct, andresearch will benefit from the inclusion of multiple measures. This is true both in thenarrow psychometric sense that error can be reduced by combining measures, as wellas in the broader sense of discovery-making when it turns out that the measures donot in fact tap the same construct.ReferencesAllen, B. P. (1975). Social distance and admiration reactions of ‘unprejudiced’ whites. Journalof Personality, 43, 709–726.Alwin, D. F. (1992). Information transmission in the survey interview: Number of responsecategories and the reliability of attitude measurement. Sociological Methodology, 22, 83–118.Alwin, D. F. (1997). Feeling thermometers versus 7-point scales: Which are better? SociologicalMethods & Research, 25, 318–340.Alwin, D. F., & Krosnick, J. A. (1991). The reliability of survey attitude measurement.The influence of question and respondent attributes. Sociological Methods & Research, 20,139–181.Anderson, B. A., Silver, B. D., & Abramson, P. R. (1988). The effects of the race of theinterviewer on race-related attitudes of black respondents in SRC/CS national electionstudies. Public Opinion Quarterly, 52, 289–324.