RESEARCH METHOD COHEN ok

TRUE EXPERIMENTAL DESIGNS 277
2 Subtract the pretest score from the post-test
score for the control group to yield score 2.
3 Subtract score 2from score 1.
Using Campbell’s and Stanley’s terminology, the
effect of the experimental intervention is:
(O 2 − RO 1 ) − (O 4 − RO 3 )
If the result is negative then the causal effect was
negative.
One problem that has been identified with this
particular experimental design is the interaction
effect of testing. Good (1963) explains that
whereas the various threats to the validity of the
experiments listed in Chapter 6 can be thought
of as main effects, manifesting themselves in
mean differences independently of the presence
of other variables, interaction effects, as their
name implies, are joint effects and may occur even
when no main effects are present. For example,
an interaction effect may occur as a result of
the pretest measure sensitizing the subjects to
the experimental variable. 1 Interaction effects
can be controlled for by adding to the pretestpost-test
control group design two more groups
that do not experience the pretest measures.
The result is a four-group design, as suggested
by Solomon (1949) below. Later in the chapter,
we describe an educational study which built into
apretest-post-testgroupdesignafurthercontrol
group to take account of the possibility of pretest
sensitization.
Randomization, Smith (1991: 215) explains,
produces equivalence over a whole range of
variables, whereas matching produces equivalence
over only a few named variables. The use
of randomized controlled trials (RCTs), a
method used in medicine, is a putative way of
establishing causality and generalizability (though,
in medicine, the sample sizes for some RCTs is
necessarily so small – there being limited sufferers
from a particular complaint – that randomization
is seriously compromised).
A powerful advocacy of RCTs for planning and
evaluation is provided by Boruch (1997). Indeed
he argues that the problem of poor experimental
controls has led to highly questionable claims being
made about the success of programmes (Boruch
1997: 69). Examples of the use of RCTs can be
seen in Maynard and Chalmers (1997).
The randomized controlled trial is the ‘gold
standard’ of many educational researchers, as
it purports to establish controllability, causality
and generalizability (Coe et al. 2000;Curriculum,
Evaluation and Management Centre 2000). How
far this is true is contested (Morrison 2001b).
For example, complexity theory replaces simple
causality with an emphasis on networks, linkages,
holism, feedback, relationships and interactivity in
context (Cohen and Stewart 1995), emergence,
dynamical systems, self-organization and an
open system (rather than the closed world
of the experimental laboratory). Even if we
could conduct an experiment, its applicability
to ongoing, emerging, interactive, relational,
changing, open situations, in practice, may be
limited (Morrison 2001b). It is misconceived to
hold variables constant in a dynamical, evolving,
fluid, open situation.
Further, the laboratory is a contrived, unreal
and artificial world. Schools and classrooms
are not the antiseptic, reductionist, analysedout
or analysable-out world of the laboratory.
Indeed the successionist conceptualization of
causality (Harré 1972), wherein researchers make
inferences about causality on the basis of
observation, must admit its limitations. One
cannot infer causes from effects or multiple causes
from multiple effects. Generalizability from the
laboratory to the classroom is dangerous, yet
with field experiments, with their loss of control
of variables, generalizability might be equally
dangerous.
Classical experimental methods, abiding by the
need for replicability and predictability, may not be
particularly fruitful since, in complex phenomena,
results are never clearly replicable or predictable:
we never step into the same river twice. In
linear thinking small causes bring small effects and
large causes bring large effects, but in complexity
theory small causes can bring huge effects and
huge causes may have little or no effect. Further,
to atomize phenomena into measurable variables
Chapter 13