Asking Questions - The Definitive Guide To Questionnaire Design ...

300 ASKING QUESTIONS
larger studies, computers can generally be more cost-effective. It is
important, however, not to evaluate your choice solely on the basis
of cost. Critical issues of sample representativeness and data quality
should also be considered. (See Sudman and Wansink, 2002, for a
more complete discussion.)
The powerful ability of the computer to execute complex skip
instructions also has some problems attached to it. The more complex
the skip instructions, the greater the possibility that an error
will creep into the programming so that questions are skipped improperly,
or that certain patterns of responses cause the computer to
freeze.
The only solution, if the complex skips are required, is to spend
the time to test how the questionnaire flows, under a wide variety
of different answer patterns. This adds to the development time of
the questionnaire, but saves much agony later. Even with substantial
testing, some bugs may remain that will not be caught until the
main survey; however, if the frequency and importance of these
bugs is very low, they may usually be ignored or corrected during the
field period.
You should also be aware that complex skips and branching possible
with CATI might make it more difficult to analyze your data
and to obtain simple margins. With complex branching, the same
variable will have different names depending on the branch taken,
and these differently named variables will need to be combined for
final analysis, which is not always a simple task.
Furthermore, conducting a survey partly on CATI and partly
with a paper-and-pencil questionnaire is messy. You cannot simply
print out the CATI questionnaire and expect the interviewer to
follow it. Also, merging data from a paper-and-pencil version with
data from CATI interviews can be tricky. Instead of planning a
mixed-mode study, think hard and creatively as to whether all
parts can be either computer-assisted such as CATI and CAPI or all
parts be done on paper. This will save you a lot of later-stage
headaches.