OTIS Evaluation Guide (PDF) - California Department of Public Health

How many waves of the survey will be conducted?
This depends on the needs of your objective. The purpose of two or more waves is to compare responses at
different points in time, such as before and after an intervention, in order to determine if it made a difference
in their attitude, knowledge, and/or behavior.
How will the sample be selected?
Most likely, the sample will be all the participants, unless the number of participants is too large (e.g., more
than 500 tobacco retailers in a very large county). In the case of a large number of participants, a random
sample should be used.
How many will be in the sample?
It will be either the total number of participants or a sample of a large number of participants.
What is the primary method in which the survey will be conducted?
In many cases, the survey can be conducted on-site immediately after the intervention. For educational
activities that do not have face-to-face contact, surveys can be mailed to the participants, but you should be
prepared for a low response rate.
How will the results be analyzed and interpreted?
The analysis of an education/participant survey will probably be quantitative (e.g. measurable, numbered)
in nature. However, if you also include open-ended questions, it can allow you to get a better idea of their
thoughts and opinions (qualitative data). A pre- and post-survey is highly desirable because you can detect
the immediate difference on attitude and intention as a result of your intervention. It is important to keep in
mind that these results describe important program attributes, but they are not the outcome for your objective.
Statistical analysis should be performed to see if there is a difference between responses before and after the
intervention/education activity. The findings can add to your understanding of the quality of the intervention/
education efforts and sometimes can demonstrate the progress of the program.
5. Policy Record (Formerly called “City or County Council” form)
The questions in the “Policy Record ” form can be seen in Figure 32.
What is the topic of the policy record you are collecting?
Obviously you are interested in, and will want to collect, all official records and reports of member voting or
discussion related to your policy objective.
How will findings from the policy record be analyzed?
A thorough content analysis should be conducted. This analysis, along with the findings from your other homework,
can be used by your coalition to identify who or what special interest groups may oppose or be behind
the opposition of the passage of the ordinance. Sometimes these results identify the messages or intervention
activities that have not yet had an impact, and they can prompt you to investigate “why?”
How will the policy record be used to advance your objective?
If any policy record collection is conducted in the middle or early stage of your funding cycle, you may use
the analysis results to modify or generate new intervention strategies. The challenge is to make sure that the
OTIS Evaluation Guide • March 2007
105