OTIS Evaluation Guide (PDF) - California Department of Public Health

and attitudes, this is not your outcome measure. Storeowner attitudes may be a useful intervening variable to
help you plan, monitor, and modify your intervention, but it is not the outcome variable directly related to your
objective. In this case, the specific outcome to be measured is the number of tobacco ads.
What type of study design will be used?
OTIS offers you three possible answers to this question:
• Experimental design
• Quasi-experimental design
• Non-experimental design
These are names that carry more value than just the melodious effect they have when recited aloud. Familiarity
with the concepts behind them will give you the opportunity to incorporate the type of rigorous evaluation
design in your plan that can best serve your objective.
• Experimental design is the most powerful and effective study design. By definition, it involves at least one
control group and at least one intervention group , as well as random assignment of participants to
these groups in the study. It is the best approach to be used in supporting any causal conclusion.
For example, if you want to see the effectiveness of your intervention on the enforcement of the STAKE Act,
and you are using the Youth Tobacco Purchase Survey, choose six demographically similar communities and
then randomly assign them to one of two groups (intervention or control.) Three communities will comprise the
intervention group (and will receive an extensive intervention), and three will comprise the control group (and
will not receive any intervention). This is a typical experimental design.
Another example would be if you wanted to assess the effectiveness of an Internet-based smoking cessation
program compared to a face-to-face cessation program. You could randomly assign smokers who called for
assistance to either the Internet or the face-to-face program and compare the difference in quit attempts or
other smoking-related behaviors.
• Quasi-Experimental designs require either a control group or multiple measurements (comparison over
time and comparison with other groups ), but DO NOT require random assignment. This is often done
when assignments have to be made based upon convenience or availability. If you have difficulties doing a
random assignment, you can match the intervention group and control group by demographic characteristics.
For example, if you have six communities in your county that are totally different in terms of population size,
you should not randomly assign them to an intervention or control group, because they are not equivalent
groups. After doing some research, perhaps you notice that two of six communities are urban regions with
a large population. Another two communities are suburban with a mid-size population, and the two remaining
communities are rural. You select one urban region, one mid-size suburban region, and one rural region
to comprise the intervention group. The other three communities are assigned to the control group. Although
this is not a random assignment, it is still a good choice because your intervention and control groups are now
similar and somewhat comparable. This is just one example of a typical quasi-experimental design.
However, in a real world setting, it may be difficult for you to decide which communities or cities should
receive the intervention and which should not. To avoid this possible ethical dilemma, you can use the socalled
“switching replications design.” In this type of quasi-experimental design, intervention activities are
OTIS Evaluation Guide • March 2007
79