In Search of Evidence

Systematic Reviews and Meta-Analyses
an indication of serious methodological limitations, which can have major
implications for the validity.
If the validity of the systematic review is satisfactory, the results can be
evaluated. If not, the results are not reliable! The reader then has to judge
whether the author's conclusions and recommendations match the level of
evidence in the studies examined. The research design of the studies in question
and the hierarchy of evidence have to be addressed again. Based on the
outcome of a meta-analysis of observational research, for instance, it is not
possible to conclude whether the studied intervention or success factor is the
cause of the effect identified; at the very most the conclusion can be drawn
that there is a correlation between these two. However, as a result of the
limited level of evidence in observational research the author cannot comment
on whether this is a causal link or on the precise nature of the cause and effect.
The same applies to a systematic review of surveys or case studies. Based on
such forms of research, it is only possible to draw conclusions about the
experience and applicability of the studied intervention or success factor, and
not about the effectiveness of these two. The author must take this into
account when formulating his conclusion (see Table 1).
Finally, the reader must judge whether the outcome of the systematic
review is generalizable and is applicable to his own management practice. In
assessing the generalizability, the reader must examine the five PICOC elements
on which the research question of the systematic review is based. The reader
will then have to assess for himself whether the result of the systematic review
of studies of the effect of participative goal setting in non-profit organizations
with a large number of highly skilled professionals, such as in a hospital, is
sufficiently generalizable to enable it to be applied to a commercial organization
with predominantly administrative staff, such as an insurance company. No allembracing
guidelines can be given for this. With reference to applicability, the
reader has to be particularly critical if no objectifiable and measurable factors
are mentioned in the conclusion and recommendations, but instead general
terms such as ‘focus’, ‘alignment’ or ‘action-driven’ are used. Although such
terms have a high generalizability because of the level of abstraction, the lack of
clarity also leads to a severe reduction in applicability. The reader will therefore
have to assess critically whether such abstractions represent adequate tools that
can be used in his/her own management practice.