Evaluating organizational stress-management interventions using ...

EVALUATION OF INTERVENTIONS 25
patterns (Colarelli, 1998; Griffiths, 1999; Hartley, 2002; Heaney, Israel,
Schurman, Baker, House, & Hugentobler, 1993). In unpredictable or
uncontrolled settings such an approach (1) raises the risk of Type III error
(erroneously concluding an intervention is ineffective when it is actually its
implementation that is faulty; Dobson & Cook, 1980) and (2) limits
explanatory yield (e.g., inconsistent intervention effects remain difficult to
explain; see Cox et al., 2000b; Parkes & Sparkes, 1998). Moreover,
controlled or predictable intervention exposure patterns occur so infrequently
that there is a need for alternative ways of managing quantitative
evaluation 1 that are viable in the face of unpredictable and uncontrollable
exposure patterns (Colarelli, 1998; Kompier, Aust, van den Berg, & Seigrist,
2000a). In summary, the identification of causal relationships may be
hindered unless study designs are adapted to reflect true, but uncontrollable
and unpredictable, patterns of intervention exposure.
ADAPTED STUDY DESIGNS AS AN EVALUATION
STRATEGY
Applied social scientists (such as those evaluating public health promotion
or large-scale community education programmes) have achieved good
results by being flexible in their application of the principles of study design
(Fitzgerald & Rasheed, 1998; Harachi, Abbot, Catalano, Haggerty, &
Fleming, 1999; Lipsey & Corday, 2000). When, for example, working in
community settings they have adapted study designs, through the use of
process evaluation, to reflect actual intervention exposure patterns (Lipsey,
1996; Lipsey & Corday, 2000). On-going or post hoc measures of
intervention exposure (i.e., process evaluation: see Kompier & Kristensen,
2000; Yin, 1994, 1995; Yin & Kaftarian, 1997) have been used to identify or
adapt the evaluation design so that the evaluation can ‘‘work backward from
the target clientele and what they actually receive/experience, not forward
from the intervention activities and what the intervention agents purportedly
deliver’’ (Lipsey, 1996, p. 301).
Given the sometimes insurmountable difficulties associated with intentionally
introducing and controlling intervention exposure, this flexible
approach to evaluation offers a practical means of evaluating stress
management interventions. Data on exposure to interventions can be
obtained through an intervention process evaluation (i.e., questioning
participants about their experiences and triangulating those data with
documentary information and by interviewing those involved in planning
and implementing interventions; Griffiths, 1999; Nytro, Saksvik, Mikkelsen,
1 The authors recognize that qualitative methods also offer viable alternative approaches to
evaluation in chaotic organizational settings (see Kompier et al., 2000a).