28.10.2014 Views

Combining health and social protection measures to reach the ultra ...

Combining health and social protection measures to reach the ultra ...

Combining health and social protection measures to reach the ultra ...

SHOW MORE
SHOW LESS

You also want an ePaper? Increase the reach of your titles

YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.

Decision-making<br />

Challenges in employing systematic reviews<br />

for <strong>health</strong> policy-making<br />

While in principle systematic reviews have much <strong>to</strong> offer <strong>the</strong><br />

policy-maker, in practice, at this point in time, <strong>the</strong>re are some<br />

real challenges in terms of <strong>the</strong>ir use in policy development.<br />

Four main challenges are described <strong>and</strong> addressed here,<br />

namely:<br />

✜ addressing non-effectiveness questions;<br />

✜ making decisions when evidence is scarce;<br />

✜ adapting international evidence <strong>to</strong> local contexts;<br />

✜ combining systematic review evidence with o<strong>the</strong>r types of<br />

information.<br />

Broadening out from effectiveness questions – over <strong>the</strong><br />

last two decades, scientists have established a reasonable<br />

consensus regarding “best practice” for systematic reviews<br />

that inform clinical decision-making regarding effectiveness,<br />

as well as public policy-making regarding adoption, use <strong>and</strong><br />

discontinuation of <strong>health</strong> technologies. These methods rely<br />

heavily on <strong>the</strong> use of hierarchies of levels of evidence, with<br />

r<strong>and</strong>omized controlled trials given <strong>the</strong> greatest weight.<br />

Statistical methods (meta-analysis) may or may not be used<br />

<strong>to</strong> analyse <strong>and</strong> summarise <strong>the</strong> results of <strong>the</strong> included studies.<br />

However, many of <strong>the</strong> questions which policy-makers raise<br />

regarding <strong>health</strong> systems do not concern whe<strong>the</strong>r or not a<br />

particular strategy works, but ra<strong>the</strong>r, how it should best be<br />

implemented, or how it might be perceived by stakeholders.<br />

Increasingly, systematic review methodologies have been<br />

applied <strong>to</strong> questions o<strong>the</strong>r than those about <strong>the</strong> effectiveness<br />

of interventions, however <strong>the</strong>re is less consensus about <strong>the</strong><br />

methodologies for conducting such reviews 3 . If systematic<br />

reviews are <strong>to</strong> be used <strong>to</strong> help policy-makers address difficult<br />

decisions regarding <strong>health</strong> systems, <strong>the</strong>n greater<br />

methodological consensus <strong>and</strong> clarity on addressing noneffectiveness<br />

questions is required.<br />

Making decisions when evidence is scarce – as noted<br />

above, systematic reviews have typically focused upon<br />

syn<strong>the</strong>sizing findings from studies that have used<br />

experimental designs. However in <strong>the</strong> field of <strong>health</strong> policy<br />

<strong>and</strong> systems research, relatively limited evidence is generated<br />

through r<strong>and</strong>omized controlled trials. In such circumstances,<br />

systematic reviews must rely more heavily on nonexperimental<br />

analyses including controlled before/after<br />

studies <strong>and</strong> interrupted time-series studies. Although<br />

methods are available for syn<strong>the</strong>sizing findings from such<br />

studies, <strong>the</strong>y are less well known <strong>and</strong> less well developed<br />

than those for syn<strong>the</strong>sizing findings from r<strong>and</strong>omized studies.<br />

But even if this broader range of study designs are included<br />

in <strong>the</strong> review, in many instances very few studies are<br />

available. For example, Lagarde <strong>and</strong> Palmer 4 conducted a<br />

systematic review of various <strong>health</strong> financing mechanisms,<br />

focusing only on studies conducted in low- <strong>and</strong> middleincome<br />

country contexts. For many of <strong>the</strong> mechanisms which<br />

<strong>the</strong>y were interested in, <strong>the</strong>re were very few studies that met<br />

<strong>the</strong>ir inclusion criteria 5 , for example no studies were found of<br />

<strong>social</strong> <strong>health</strong> insurance mechanisms, one was found for<br />

community-based <strong>health</strong> insurance, <strong>and</strong> three for contracting<br />

out of <strong>health</strong> services. When so few studies are available, <strong>the</strong><br />

information syn<strong>the</strong>sized can be frustratingly thin, <strong>and</strong> not very<br />

helpful <strong>to</strong> policy-makers. Systematic reviews may still be<br />

useful, <strong>to</strong> at least point out <strong>the</strong> flimsy nature of <strong>the</strong> evidence<br />

base supporting <strong>the</strong> policy or intervention being considered,<br />

<strong>and</strong> thus underscoring <strong>the</strong> need for <strong>the</strong> simultaneous<br />

implementation of moni<strong>to</strong>ring <strong>and</strong> evaluation mechanisms <strong>to</strong><br />

ensure that no harm is done through <strong>the</strong> policy. However this<br />

challenge also implies: (a) that much greater investment is<br />

needed in well-designed, rigorous, impact evaluations <strong>and</strong> (b)<br />

that development of methods for <strong>the</strong> syn<strong>the</strong>sis of o<strong>the</strong>r types<br />

of study design should be accelerated, while acknowledging<br />

that, depending on <strong>the</strong> nature of <strong>the</strong> question being asked,<br />

<strong>the</strong>y may provide less reliable information.<br />

Adapting international evidence <strong>to</strong> local contexts – most<br />

systematic reviews review <strong>the</strong> international evidence base,<br />

yet, as described above, <strong>the</strong>re are major differences in <strong>the</strong><br />

<strong>health</strong> systems of different countries, <strong>and</strong> thus <strong>the</strong> effects that<br />

alternative strategies <strong>to</strong> streng<strong>the</strong>n <strong>health</strong> systems might lead<br />

<strong>to</strong>. For example, Lewin et al 6 review <strong>the</strong> evidence on lay<br />

<strong>health</strong> workers. While <strong>the</strong>y found many articles 7 that met <strong>the</strong>ir<br />

inclusion criteria, <strong>and</strong> <strong>the</strong>y concluded that <strong>the</strong>re was evidence<br />

of moderate <strong>to</strong> high quality supporting <strong>the</strong> effectiveness of lay<br />

<strong>health</strong> workers in <strong>the</strong> provision of specific services such as<br />

improving immunization uptake, <strong>and</strong> reducing childhood<br />

morbidity <strong>and</strong> mortality from common illnesses, <strong>the</strong> authors<br />

noted that for some services much of <strong>the</strong> available evidence<br />

came from industrialized countries <strong>and</strong> it was not clear<br />

whe<strong>the</strong>r <strong>the</strong> intervention effects would be transferable <strong>to</strong> o<strong>the</strong>r<br />

settings. With respect <strong>to</strong> <strong>the</strong> effectiveness of lay <strong>health</strong><br />

workers, differences in contextual fac<strong>to</strong>rs such as <strong>the</strong><br />

availability of routine data, <strong>the</strong> availability of resources <strong>to</strong><br />

provide clinical <strong>and</strong> managerial support, <strong>and</strong> <strong>the</strong> availability<br />

of drugs <strong>and</strong> accessible referral services are all likely <strong>to</strong> be<br />

critical. While some systematic reviews (e.g. Lagarde <strong>and</strong><br />

Palmer 2007) focus on low- <strong>and</strong> middle-income country<br />

contexts alone, it is not clear that focusing only on studies<br />

from countries with a particular economic status necessarily<br />

leads <strong>to</strong> more relevant conclusions. Improved approaches <strong>to</strong><br />

ensuring <strong>and</strong> assessing <strong>the</strong> transferability of review findings<br />

are needed.<br />

Faced with a review that syn<strong>the</strong>sizes evidence from a<br />

variety of potentially very different country contexts, policymakers<br />

need <strong>to</strong> question whe<strong>the</strong>r <strong>the</strong>re are important relevant<br />

differences in <strong>the</strong> structure of <strong>the</strong> <strong>health</strong> system, <strong>the</strong> on-<strong>the</strong>ground<br />

realities <strong>and</strong> constraints, <strong>the</strong> epidemiological<br />

conditions, or <strong>the</strong> perspectives of <strong>health</strong> system stakeholders,<br />

that might mean that <strong>the</strong> review findings are not transferable<br />

<strong>to</strong> <strong>the</strong> context where <strong>the</strong>y are being considered. Policymakers,<br />

or policy analysts both need sufficient skills <strong>to</strong> make<br />

this assessment <strong>and</strong> be provided with sufficient information<br />

within <strong>the</strong> review for <strong>the</strong>m <strong>to</strong> be able <strong>to</strong> assess transferability.<br />

A fur<strong>the</strong>r fac<strong>to</strong>r regarding <strong>the</strong> transferability of findings<br />

regards <strong>the</strong> scale at which <strong>the</strong> intervention was implemented;<br />

often systematic reviews of effectiveness questions capture<br />

data from impact evaluations of small scale, experimental<br />

interventions. Some interventions may be effective if<br />

Global Forum Update on Research for Health Volume 4 ✜ 149

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!