Question and Questionnaire Design - Stanford University

Question and Questionnaire Design 281effects (Carp, 1974; Chan, 1991; Holmes, 1974; Johnson, 1981; Payne, 1971; Quinn &Belson, 1969). Furthermore, the two studies of rating scales that used oralpresentation found primacy effects as well (Kalton, Collins, & Brook, 1978; Mingay& Greenwell, 1989). Consistent with the satisficing notion, Mingay and Greenwell(1989) found that their primacy effect was stronger for people with more limitedcognitive skills. However, these investigators found no relation of the magnitude ofthe primacy effect to the speed at which interviewers read questions, despite the factthat a fast pace presumably increased task difficulty. Also, response order effectswere no stronger when questions were placed later in a questionnaire (Carp, 1974).Thus, the moderators of rating scale response order effects may be different from themoderators of such effects in categorical questions, though more research is clearlyneeded to fully address this question.How should researchers handle response order effects when designing surveyquestions? One seemingly effective way to do so is to counterbalance the order inwhich choices are presented. Counterbalancing is relatively simple to accomplishwith dichotomous questions; a random half of the respondents can be given oneorder, and the other half can be given the reverse order. When the number ofresponse choices increases, the counterbalancing task can become more complex.However, when it comes to rating scales, it makes no sense to completely randomizethe order in which scale points are presented, because that would eliminate thesensible progressive ordering from positive to negative, negative to positive, most toleast, least to most, etc. Therefore, for scales, only two orders ought to be used,regardless of how many points are on the scale.Unfortunately, counterbalancing order creates a new problem: variance in responsesdue to systematic measurement error. Once response alternative orders have beenvaried, respondent answers may differ from one another partly because different peoplereceived different orders. One might view this new variance as random error variance,the effect of which would be to attenuate observed relations among variables and leavemarginal distributions of variables unaltered. However, given the theoreticalexplanations for response order effects, this error seems unlikely to be random.Thus, in addition to counterbalancing presentation order, it seems potentiallyvaluable to take steps to reduce the likelihood of the effects occurring in the first place.The most effective method for doing so presumably depends on the cognitivemechanism producing the effect. If primacy effects are due to satisficing, then steps thatreduce satisficing should reduce the effects. For example, with regard to motivation,questionnaires can be kept short, and accountability can be induced by occasionallyasking respondents to justify their answers. With regard to task difficulty, the wordingof questions and answer choices can be made as simple as possible.9.6. Treatment of No-OpinionWhat happens when people are asked a question about which they have no relevantknowledge? Ideally, they will say that they do not know the answer. But respondentsmay wish not to appear uninformed and may therefore give an answer to satisfy the