Evaluating organizational stress-management interventions using ...

More documents

Recommendations

Info

36 RANDALL, GRIFFITHS, COX involved in the intervention reported lower exhaustion scores than the group not involved in the intervention, but this difference only approached significance, F(1, 30) = 2.24, p = .14. Taken together, however, the effect of the changes within the two groups was significant: Interaction terms are more sensitive than separate withingroups analysis (Tabachnick & Fidell, 2001). As in Study 1, these results suggested that exposure to the intervention impacted on participants’ wellbeing: This was only apparent when measures of exposure were used to adapted the design of the study during analysis. DISCUSSION The significant Measured exposure 6 Time interaction effects in both studies indicated that exposure to the intervention predicted changes in exhaustion scores over time. The central hypothesis of this article was confirmed: Nonadaptive designs underestimated the impact of the intervention on well-being with significant change only becoming apparent when adapted study designs were used. Both studies indicated that Type III error (Dobson & Cook, 1980; Harachi et al., 1999) may be minimized by using the results of a robust process evaluation (in this study one that centred on the triangulation of exposure data) to adapt outcome evaluation. This protection against Type III error is particularly important given that the psychological components of work design may exert a relatively modest influence over general well-being in the short term (Zapf, Dormann, & Frese, 1996). Moreover, in both studies measures of intervention exposure identified hidden and ‘‘unintended’’ between-groups designs that facilitated much-needed and significant improvements in methodological adequacy (see Beehr & O’Hara, 1987; Murphy, 1996). In Study 1, the between-group differences found at Time 2 reflected a worsening of the situation for the ‘‘not aware’’ group cooccurring with stability in the ‘‘aware’’ group, suggesting that the return of responsibilities to them protected supervisors from the effects of problems associated with not being able to report faults. This ‘‘protective effect’’ has been observed in other intervention studies (e.g., Terra, 1995). The pattern of change in Study 2 indicated a significant intervention effect when the small changes in the intervention and control groups cooccurred. Clearly, and for good reasons, the methodological adequacy of both studies reflected the constraints placed on them by the research setting. Like almost all stress management intervention evaluation studies they do not play exactly by the methodological rules (Kompier et al., 2000a). However, measuring and capitalizing on uncontrolled and unpredictable exposure patterns did extend the established principles of the ‘‘natural experiment’’. Indeed, in the majority of situations where complete control over
EVALUATION OF INTERVENTIONS 37 intervention exposure is not possible, this approach offers a practical and informative method of evaluation. Of course, it should only be used when rigorous process evaluation yields a strong case for adapting the study design: The methodology should not be used as a justification for ‘‘fishing’’ in data for significant results (Yin, 1994). The small sample sizes provided by the majority of organizations (as was the case in both of the studies presented here) make theory-driven analysis and preanalysis checks on the data particularly important. Naturally, this approach is not without its problems. Unpredictable exposure patterns create an extreme form of nonequivalent study design and may be affected by selection biases (Campbell & Stanley, 1963; Cook & Campbell, 1979). In both of the studies presented here, intervention and control groups were comparable across a range of demographic variables, and appeared well-matched at Time 1 on the measure of the dependent variable. However, when this is not the case possible bias can be controlled for by the analysis of covariates, or by using ‘‘blocking’’ designs that match cases on an ad hoc basis (Cook & Campbell, 1979; Cook & Shadish, 1994). Further, the list of active ingredients (Kompier & Kristensen, 2000) that are indicative of intervention exposure across a range of different types of intervention is not yet established. Measuring involvement may be appropriate for interventions that require the participant to actively engage in the intervention for it to work (e.g., a training intervention, teambuilding intervention, staff consultation process, etc.). Measuring awareness may be more appropriate for interventions that have a more passive mechanism (e.g., the receipt of information, changes in guidelines, or a redefinition of roles and responsibility). Further work is needed to establish the modus operandi of a variety of organizational interventions in order to ensure that appropriate measures of intervention exposure are used to adapt study designs. Triangulating the self-report data using process evaluation helped to guard against threats to the validity of the self-report intervention exposure measure, and greatly enhanced understanding of the intervention effects. In Study 1, it appeared that staff who were managed by particular senior managers had not received the intervention. In Study 2, exposure to the intervention was determined by the speed at which the hospital’s computer technicians could install and make fully operational the computer equipment. These findings indicated that reported exposure to the intervention was driven by implementation factors rather than by any individual differences (e.g., negative affect: Brief, Burke, George, Robinson, & Webster, 1988; Watson & Clark, 1984) or methodological artifacts (e.g., common method variance; Spector, 1994). Even in controlled exposure studies such data about the change process should be gathered to enhance the explanatory yield of outcome evaluation (Cook & Shadish, 1994).
Page 1 and 2: EUROPEAN JOURNAL OF WORK AND ORGANI
Page 3 and 4: EVALUATION OF INTERVENTIONS 25 patt
Page 5 and 6: EVALUATION OF INTERVENTIONS 27 base
Page 7 and 8: EVALUATION OF INTERVENTIONS 29 expe
Page 9 and 10: EVALUATION OF INTERVENTIONS 31 F(1,
Page 11 and 12: EVALUATION OF INTERVENTIONS 33 TABL
Page 13: EVALUATION OF INTERVENTIONS 35 Figu
Page 17 and 18: EVALUATION OF INTERVENTIONS 39 Camp
Page 19: EVALUATION OF INTERVENTIONS 41 Saks

Evaluating organizational stress-management interventions using ...

Create successful ePaper yourself

Delete template?

Save as template?