OTIS Evaluation Guide (PDF) - California Department of Public Health

provided to Group A, with no intervention in Group B. After a certain time period, the same intervention
activities are delivered to Group B, with no intervention in Group A. Thus, the original “intervention group” and
“control group” actually switch their roles. Measurements are made at the beginning of the program, before the
switching, and after the switching at the end of the program.
• Non-experimental designs have the least capability to reflect or demonstrate the success of a program.
This type of design does not use control groups or random assignment; therefore, at best, it can provide only
a weak indication of a possible connection between intervention and outcome. However, it is useful in many
situations when a stronger design is not applicable or the budget is not sufficient.
Let’s look at an example that is not clear-cut in order to illustrate the rationale for determining the appropriate
study design. One of the most commonly used designs is a pre- and post-test only with one intervention group.
Difference or change can be detected by comparing the pre- and post-test results. Is this a quasi-experimental
or a non-experimental design? This question is still under debate.
80
• In our opinion, if only one set of pre- and post-tests is conducted, the design should be classified as
non-experimental. This is because any number of unanticipated events other than the intervention activities
may also produce the desired change, but the design is not capable of detecting their influence.
Therefore, extraneous factors cannot be controlled for in the analysis.
• However, if there is more than one pre-test and more than one post-test, the design should be called
quasi-experimental because it uses multiple measurements before, during and after the intervention. In
this situation you virtually treat the only group as both intervention and control group. Any anecdotal
event (other than the intervention activities) that results in a change may be detected during the multiple
measurements. You can determine when the anecdotal event and change occurred, and control for it in
the analysis to determine if the intervention has contributed to this change.
Groups
If you answered “experimental or quasi-experimental” in Figure 19, OTIS requests information about the number
of intervention and/or control groups, and whether your design contains an intact group (Figure 19):
Number of Intervention Groups and Number of Control Groups:
To answer this question, consider this scenario: 40 bars in three cities will receive the intervention and 40 bars
in another three cities will receive no intervention. What do you enter for the number of intervention groups
and the number of control groups? 40? 3?
The answer is “1” intervention group and “1” control group. Multiple intervention groups only exist when
multiple, different intervention strategies are given to different communities. If each of these communities has
its own control community, then we have multiple control groups as well.
Intact group:
This strategy is rooted in the same rationale for using intervention groups and control groups, that is, to
increase comparability among different groups. An intact group is the same group of people who are
measured with both a pre-test and a post-test. An advantage to using intact groups for experimental or quasi-
OTIS Evaluation Guide • March 2007