Systematic Review - Network for Business Sustainability

The studies in our sample are mixed in their technical
accomplishment, with an approximately equal number
of well- and poorly-executed studies. A little over
a quarter of the studies did not report in sufficient
detail to allow us to judge their level of technical
accomplishment.
To assess the strength of evidence of included
studies, we adopted Reay, Berta & Kohn’s (2009)
rubric, modified from a model for evaluating the
quality of medical research (Table 10). Their hierarchy
Table 10
HIERARCHY OF STRENGTH OF EVIDENCE
(Source Reay et al., 2009)
Level 1 evidence (strongest) is generated through…
• RCTs [randomized controlled trials] or meta-analyses
Level 2 evidence emerges from…
• (a) A high-quality literature review that is replicable and
comprehensive and provides a synthesis and actionable
recommendations based on the synthesis or (b) a
systematic literature review.
Level 3 evidence is garnered through…
• Comparative, multisite case studies or large-sample
quantitative studies involving data collected from more
than one site (organization).
Level 4 evidence is gathered through…
• Small-sample, single-site qualitative or quantitative
studies. These studies are theoretically motivated and are
completed by trained (management) researchers who have
(at most) an arm’s-length relationship with the organization
under study; the “voice” of these studies is objective.
Level 5 evidence is generated through…
• Descriptive studies and/or self-report stories. These
studies generally include observations, admonitions, and
recommendations relevant to managers. Early papers
important to the then “new” area of evidence-based
management offered emerging theory bolstered by Level
5 evidence.
Level 6 evidence (weakest) is based on…
• The opinion of respected authorities or expert committees
without additional data. Some papers offer anecdotal
evidence as a means of supporting such opinions.
differentiates levels according to the strength of
evidence generated.
The distribution of the studies included in this review
according to Reay et al.’s (2009) classification of levels
of evidence is presented in Figure 13 and shows our
sample to consist exclusively of evidence of levels 3, 4
and 5. The strongest level of evidence was level 3 and
the weakest, level 5: the most frequently referenced
level was level 4.
Figure 13
STRENGTH OF EVIDENCE
Innovating for Sustainability 83