Preventing Childhood Obesity - Evidence Policy and Practice.pdf

Chapter 19
Box 19.1 Methodological debate:
polarization or pragmatism?
• Historically, there has been debate over the relative
merits of quantitative and qualitative research
methods in the evaluation of social interventions.
• The period from the late 1960s to the early 1980s
was a “ golden age of evaluation ” with 245 “ randomized
field experiments ” conducted in areas such
as criminal justice, social welfare, education and
legal policy. 5
• Pragmatic mixed method approaches, where
methods or combinations of methods are pragmatically
chosen to address the specific research question,
6 has been lacking but is now developing.
• Public health is necessarily cross - disciplinary requiring
the combination and integration of research
methods from a diversity of contributing disciplines. 7
• More recently, there has been a call for a transdisciplinary
science approach using a shared
con ceptual framework to draw together the most
rig orous and appropriate discipline - specific theories,
models, methods and measures for the question
being posed. 8
A further lesson to be learned from experience elsewhere
is that the term “ evaluation ” covers a wide
range of activities, which vary greatly across a number
of dimensions. While this chapter focuses on the evaluation
of community interventions, within that focus,
it is important to recognize that evaluation projects
will vary according to the purpose of the evaluation,
the resources available to conduct the evaluation, and
the complexity of the intervention to be evaluated. We
consider each of these three dimensions, with a
primary focus on the evaluation of complex community
interventions, and the key stages in the evaluation
of such interventions.
Evaluation: p urpose and r esources
In planning any evaluation, it is important to consider
why that evaluation is taking place. Many evaluations,
particularly those carried out by practitioners rather
than researchers, are undertaken primarily as an exercise
in accountability, with an emphasis on documenting
or measuring what happened, with possibly
some attempt to identify the impact, of a particular
funded activity. Such evaluations are of limited scope
and are not really the concern of this chapter, as
they are more appropriately conducted within a
project management or performance assessment
framework, rather than being considered as evaluative
activities. Any true evaluation should aim to produce
learning and/or improvement. A good professional
ethic requires that lessons are learned regarding
the process and impact of an intervention and that
there is continuous ongoing assessment of whether
the intervention is working as anticipated and having
the desired outcomes. It is critical that the possibility
that interventions can do harm is not rejected. Many
well - intentioned interventions have been found to
be doing more harm than good in terms of their
main purpose, 1 while others have unanticipated
impacts or are detrimental to subgroups of the target
population. It is also important that professionals
strive to improve the quality of interventions, whether
by improving their reach, effectiveness, efficiency or
equity.
What i s e valuation?
There has been much debate as to the definition of
“ evaluation ”, and how it is distinct from “ research ”.
Shaw 2 proposes a three - level taxonomy, in which
“ evaluation ” (which we refer to as practitioner evaluation)
is characterized by a focus on practical problems
with the objective of informing practice
immediately and locally. It is usually undertaken by
practitioners with little emphasis on scientific rigor,
an enhanced form of reflexive professional practice.
“ Evaluation research ” uses stronger methods and
seeks to have an impact on practice to improve effectiveness
and efficiency, with dissemination through
professional and policy networks and in the grey literature.
And Shaw ’s third level is “applied research ”,
which is led by researchers using strong methods and
is disseminated through peer - reviewed scientific
papers with the aim of producing generalizable knowledge
with an impact on theory and practice over a
long - term period.
This chapter adopts a definition of evaluation in
line with Pawson and Tilley, 3 who see the purpose of
evaluation “ as informing the development of policy
and practice ” 4 rather than focusing simply on measurement
or increased understanding.
158