Preventing Childhood Obesity - Evidence Policy and Practice.pdf

Chapter 19
The final stage of formative evaluation is to conduct
an exploratory trial, in which a final draft or Beta - test
version of the intervention is tested in a small - scale
summative evaluation. This is not designed or powered
to identify an estimate of intervention effectiveness,
but is intended to further test feasibility and acceptability,
identify any important remaining barriers or
problems that need to be addressed, and to allow
testing and estimation of key components of the summative
evaluation methodology, such as outcome
measurement, recruitment and retention rates. In
many cases, the intervention development, pilot
testing and exploratory trial phases may indicate that
the intervention is not acceptable to the target population
or to stakeholders in potential future implementation,
or that there is no evidence that it is bringing
about the anticipated changes and effects. This will
lead to further modification or abandonment of the
intervention. In other cases, the intervention will
appear to be feasible, acceptable and potentially effective,
and will thus be ready for large - scale summative
evaluation.
Summative e valuation
Once a thorough process of formative evaluation has
been completed, an important remaining question,
which for most policy decision makers is the most
important question, is whether or not the intervention
works. This requires an estimate of the intervention ’ s
effect, which may then be compared to that of competing
interventions, often using some form of economic
analysis trading off costs and benefits of
alternative programs. Pawson 20 argues that, in the case
of complex social programs, it is futile to attempt an
experimental evaluation, comparing a “ policy on ”
condition with a “ policy off ” condition, since such
programs are constantly being manipulated and renegotiated
and are never stable. With such large - scale
complex programs, Pawson states that “ the hallowed
comparison of treatment and controls is … that
between a partial and complete mystery ” .
It is undoubtedly the case that many large - scale
interventions are not feasibly or sensibly evaluated
using experimental or quasi - experimental control
group designs. This will generally be the case for mass
media interventions applied to whole populations,
and to large - scale complex social programs that will
be substantially modified during the course of their
evaluation. It is also unlikely to be worth while implementing
a scientifically rigorous research design of an
intervention that is so politically contentious that its
evaluation is “ doomed to success ” , the political cost
of a negative evaluation being too high. 21
Efficacy and e ffectiveness
However, it is also very important to recognize that it
is not necessary for an intervention to be highly standardized
and uniformly delivered in all instances, in
order for a valuable experimental summative evaluation
to take place. This would be the case in an efficacy
trial, which seeks to identify the impact of the intervention
when delivered in ideal circumstances.
However, since health promotion interventions are so
dependent on context adaptation and the quality of
delivery, the value of efficacy trials is limited. 22 For
example, smoking education interventions have been
found to work well in efficacy trials, when delivered
by enthusiastic teachers with ample curriculum time,
yet when implemented in the real world they have not
been found to be effective. 23 In an effectiveness
trial, a pragmatic approach is taken, with the intervention
delivered in the trial in the same (variable)
way as would realistically be achieved in the real
world.
It is often argued that health promotion interventions
are so dependent on the way they are implemented,
and upon the context (environment, policy
etc.) within which they are delivered, that context -
dependent adaptation is crucial to maximize effectiveness
and, therefore, randomized controlled trials are
not suited to their evaluation. However, the RCT
design actually has the advantage that the randomization
process ensures that systematic differences in
external influences between groups do not occur
and thereby ensures that an unbiased estimate of the
average effect of the intervention is obtained. It is,
however, crucially important in an effectiveness trial
of a complex community intervention to conduct a
comprehensive qualitative investigation within the
trial, so that these variable factors can be monitored.
Thus, the qualitative research provides information
on the factors that support or attenuate the effectiveness
of the intervention. To undertake a trial of a
complex intervention without an embedded qualitative
process evaluation would be to treat the interven-
162