Question and Questionnaire Design - Stanford University

276 Jon A. Krosnick and Stanley Presserquestions. When people are given the choices ‘‘agree’’ and ‘‘disagree,’’ are not toldthe statements to which they apply, and are asked to guess what answers anexperimenter is imagining, ‘‘agree’’ is chosen much more often than ‘‘disagree’’(e.g., Berg & Rapaport, 1954). When people are asked to agree or disagree with pairsof statements stating mutually exclusive views (e.g., ‘‘I enjoy socializing’’ vs. ‘‘I don’tenjoy socializing’’), the between-pair correlations are negative but generally veryweakly so (Krosnick and Fabrigar report an average correlation of only –0.22 across41 studies). Although random measurement error could cause the correlations todepart substantially from –1.0, acquiescence could do so as well.Consistent with this possibility, averaging across 10 studies, 52% of people agreedwith an assertion, whereas only 42% of people disagreed with the opposite assertion(Krosnick & Fabrigar, forthcoming). Another set of eight studies compared answersto agree/disagree questions with answers to forced choice questions where the orderof the views expressed by the response alternatives was the same as in the agree/disagree questions. On average, 14% more people agreed with an assertion thanexpressed the same view in the corresponding forced choice question. In seven otherstudies, an average of 22% of the respondents agreed with both a statement and itsreversal, whereas only 10% disagreed with both. Thus, taken together, these methodssuggest an acquiescence effect averaging about 10%.Other evidence indicates that the tendency to acquiesce is a general inclinationof some individuals across questions. The cross-sectional reliability of the tendencyto agree with assertions averaged 0.65 across 29 studies. And the over-timeconsistency of the tendency to acquiesce was about 0.75 over one month, 0.67 overfour months, and 0.35 over four years (e.g., Couch & Keniston, 1960; Hoffman,1960; Newcomb, 1943).Similar results (regarding correlations between opposite assertions, endorsementrates of items, their reversals, and forced choice versions, and so on) have beenproduced in studies of true/false questions and of yes/no questions, suggesting thatacquiescence is present in responses to these items as well (see Krosnick & Fabrigar,forthcoming). And there is other such evidence regarding these response alternatives.For example, people are much more likely to answer yes/no factual questionscorrectly when the correct answer is ‘‘yes’’ than when it is ‘‘no’’ (e.g., Larkins &Shaver, 1967; Rothenberg, 1969), presumably because people are biased towardsaying ‘‘yes.’’Acquiescence is most common among respondents who have lower social status(e.g., Gove & Geerken, 1977; Lenski & Leggett, 1960), less formal education (e.g.,Ayidiya & McClendon, 1990; Narayan & Krosnick, 1996), lower intelligence (e.g.,Forehand, 1962; Hanley, 1959; Krosnick, Narayan, & Smith, 1996), lower cognitiveenergy (Jackson, 1959), less enjoyment from thinking (Messick & Frederiksen, 1958),and less concern to convey a socially desirable image of themselves (e.g., Goldsmith,1987; Shaffer, 1963). Also, acquiescence is most common when a question is difficult(Gage, Leavitt, & Stone, 1957; Hanley, 1962; Trott & Jackson, 1967), whenrespondents have become fatigued by answering many prior questions (e.g., Clancy &Wachsler, 1971), and when interviews are conducted by telephone as opposed to faceto-face(e.g., Calsyn, Roades, & Calsyn, 1992; Holbrook, Green, & Krosnick, 2003).