OTIS Evaluation Guide (PDF) - California Department of Public Health

Describe the activities and resources to be used to design the above instruments
If the instruments you want to use are nowhere to be found, and you have to create one yourself, you are
faced with a difficult job. If you are not a survey design specialist, please do not hesitate to use an existing
one or contact an Evaluation Associate at the TC Evaluation Center, who may be able to recommend questions
from various sources to best meet your needs. Remember, a poorly developed questionnaire might waste
a large portion of your budget. Pilot testing should be done before any instrument is formally used.
Methods for Collecting Data
Once you have described your data collection instruments, OTIS asks you to describe your methods for collecting
the data with the following two questions (see Figure 23):
• Which is the primary method in which the outcome data will be collected?
• What is the primary source where the outcome data will be collected?
Figure 23
Which is the primary method in which the outcome data will be collected?
Common methods for collecting data include: telephone surveys, mail surveys, group-administered surveys,
face-to- face interviews, and observations. Each method has its pros and cons.
• Mail surveys are very inexpensive, but tend to have very low response rates (from 10% to 30%)
unless intensive follow-up is applied. Because there are no instructions from an interviewer or instructor,
as in telephone surveys and face-to-face surveys, the quality may be compromised. The pros of this
method include giving respondents more time to think about their answers and enables you to collect
data from a large group of people from a wide geographical area.
• Written questionnaires are the most economical way to conduct a survey when your sample size is
not too large, especially if it is self-administrated. All you need are some paper and pens. However, this
approach may compromise the representativeness of the sample.
• Face-to-face surveys are very useful for complicated questions where probing by the interviewer
and/or clarification is needed by both parties. They may also be necessary to collect information from
subjects with low literacy or other special needs. Unlike written questionnaires, interviewers record the
answers from the respondents. Travel and training are associated costs to be considered.
• Telephone surveys: Computerized programs are often utilized to conduct telephone interviews.
However, a computer assisted telephone survey is often costly, especially when you are trying to get a
randomly selected sample. If you decide to do telephone surveys without the help of computer programs
and the sample size is manageable, this method can be less expensive than face-to-face surveys.
The response rate is better than a mail survey but may be worse than face-to-face surveys.
OTIS Evaluation Guide • March 2007
85