OTIS Evaluation Guide (PDF) - California Department of Public Health

experimental designs is that you can account for and therefore remove various participant characteristics
that existed before and after an intervention. As a result, you are better able to determine if in fact it was the
intervention that made an impact on your target group.
Measurements and study design narrative
The next two questions discussed will be the last ones required for this section (Figure 19):
• Measurements: (drop-down menu)
• Describe your study design: (text box)
Measurements:
Measurements are also crucial to your evaluation design. Pre- and post-tests are commonly used, and we also
recommend using multiple measurements for certain conditions, such as when your intervention is time-sensitive.
• For example, smoking cessation programs can yield a quit rate as high as 40-50% at the end of an
intervention, but only a 25% quit rate when measured one year following the intervention. These results
are determined by multiple post-tests.
• Measurements themselves can also be very sensitive. For example, the compliance rate of a smokefree
bar law may be susceptible to numerous factors. If a police officer visits a bar several times within
one week because of a high number of bar fights, compliance with the smoke free bar law may
increase dramatically. In this case, the presence of the police officer is an influencing extraneous factor.
• Multiple measurements can improve the quality of your evaluation by increasing the analysis power.
If you have the time and resources, you should do multiple measurements prior to, during, and after the
intervention. In this way, you can see at what point the change in the compliance rate occurred, and
then decide if the intervention has contributed to this change.
• Multiple measurements are especially useful when you have only one intervention group and
no control group.
Describe your study design:
We strongly encourage you to use designs that go beyond a simple one-group pre- and post-test design in
order to strengthen the value of the findings to others and to contribute answers to the “what works?” question.
You have the option to describe your design in more detail in the text box.
This completes the information for the study design. After you submit the answer, OTIS will lead you back to
the “Scope of Work/View Objective” page. As you can tell from Figure 20, the study design information you
entered appears under the subheading “Collection of Outcome Data Design”.
OTIS Evaluation Guide • March 2007
81