Combining health and social protection measures to reach the ultra ...

Decision-makingChallenges in employing systematic reviewsfor health policy-makingWhile in principle systematic reviews have much to offer thepolicy-maker, in practice, at this point in time, there are somereal challenges in terms of their use in policy development.Four main challenges are described and addressed here,namely:✜ addressing non-effectiveness questions;✜ making decisions when evidence is scarce;✜ adapting international evidence to local contexts;✜ combining systematic review evidence with other types ofinformation.Broadening out from effectiveness questions – over thelast two decades, scientists have established a reasonableconsensus regarding “best practice” for systematic reviewsthat inform clinical decision-making regarding effectiveness,as well as public policy-making regarding adoption, use anddiscontinuation of health technologies. These methods relyheavily on the use of hierarchies of levels of evidence, withrandomized controlled trials given the greatest weight.Statistical methods (meta-analysis) may or may not be usedto analyse and summarise the results of the included studies.However, many of the questions which policy-makers raiseregarding health systems do not concern whether or not aparticular strategy works, but rather, how it should best beimplemented, or how it might be perceived by stakeholders.Increasingly, systematic review methodologies have beenapplied to questions other than those about the effectivenessof interventions, however there is less consensus about themethodologies for conducting such reviews 3 . If systematicreviews are to be used to help policy-makers address difficultdecisions regarding health systems, then greatermethodological consensus and clarity on addressing noneffectivenessquestions is required.Making decisions when evidence is scarce – as notedabove, systematic reviews have typically focused uponsynthesizing findings from studies that have usedexperimental designs. However in the field of health policyand systems research, relatively limited evidence is generatedthrough randomized controlled trials. In such circumstances,systematic reviews must rely more heavily on nonexperimentalanalyses including controlled before/afterstudies and interrupted time-series studies. Althoughmethods are available for synthesizing findings from suchstudies, they are less well known and less well developedthan those for synthesizing findings from randomized studies.But even if this broader range of study designs are includedin the review, in many instances very few studies areavailable. For example, Lagarde and Palmer 4 conducted asystematic review of various health financing mechanisms,focusing only on studies conducted in low- and middleincomecountry contexts. For many of the mechanisms whichthey were interested in, there were very few studies that mettheir inclusion criteria 5 , for example no studies were found ofsocial health insurance mechanisms, one was found forcommunity-based health insurance, and three for contractingout of health services. When so few studies are available, theinformation synthesized can be frustratingly thin, and not veryhelpful to policy-makers. Systematic reviews may still beuseful, to at least point out the flimsy nature of the evidencebase supporting the policy or intervention being considered,and thus underscoring the need for the simultaneousimplementation of monitoring and evaluation mechanisms toensure that no harm is done through the policy. However thischallenge also implies: (a) that much greater investment isneeded in well-designed, rigorous, impact evaluations and (b)that development of methods for the synthesis of other typesof study design should be accelerated, while acknowledgingthat, depending on the nature of the question being asked,they may provide less reliable information.Adapting international evidence to local contexts – mostsystematic reviews review the international evidence base,yet, as described above, there are major differences in thehealth systems of different countries, and thus the effects thatalternative strategies to strengthen health systems might leadto. For example, Lewin et al 6 review the evidence on layhealth workers. While they found many articles 7 that met theirinclusion criteria, and they concluded that there was evidenceof moderate to high quality supporting the effectiveness of layhealth workers in the provision of specific services such asimproving immunization uptake, and reducing childhoodmorbidity and mortality from common illnesses, the authorsnoted that for some services much of the available evidencecame from industrialized countries and it was not clearwhether the intervention effects would be transferable to othersettings. With respect to the effectiveness of lay healthworkers, differences in contextual factors such as theavailability of routine data, the availability of resources toprovide clinical and managerial support, and the availabilityof drugs and accessible referral services are all likely to becritical. While some systematic reviews (e.g. Lagarde andPalmer 2007) focus on low- and middle-income countrycontexts alone, it is not clear that focusing only on studiesfrom countries with a particular economic status necessarilyleads to more relevant conclusions. Improved approaches toensuring and assessing the transferability of review findingsare needed.Faced with a review that synthesizes evidence from avariety of potentially very different country contexts, policymakersneed to question whether there are important relevantdifferences in the structure of the health system, the on-thegroundrealities and constraints, the epidemiologicalconditions, or the perspectives of health system stakeholders,that might mean that the review findings are not transferableto the context where they are being considered. Policymakers,or policy analysts both need sufficient skills to makethis assessment and be provided with sufficient informationwithin the review for them to be able to assess transferability.A further factor regarding the transferability of findingsregards the scale at which the intervention was implemented;often systematic reviews of effectiveness questions capturedata from impact evaluations of small scale, experimentalinterventions. Some interventions may be effective ifGlobal Forum Update on Research for Health Volume 4 ✜ 149