(CASAC) Peer Review of EPA's Integrated Science Assessment

Dr. Jonathan M. SametComments on SOx 2nd Draft ISAGeneral Comments:Overview:This second draft Integrated Science Assessment (ISA) has been redrafted following theinitial review by CASAC. Along with the draft ISA for Nitrogen Oxides, it representsthe second use of the new evidence review approach by the Environmental ProtectionAgency. As such, it remains critical that a useful model be established for future reviews.In this regard, I still find the second draft ISA to be deficient. While changes have beenmade, the approach to evidence identification, review, and synthesis remainsinadequately specified. In the revised Chapter 1, while the Agency has begun to moreformally set out criteria for evidence evaluation and standard approaches for classifyingthe strength of evidence for causation, the overall approach still needs development andits application in the subsequent chapters of the ISA is not sufficiently thoughtful. Myresponses to the charge questions follow:Question 1:This question asks whether the framework set out in Chapter 1 is appropriate andadequately applied, and whether the Agency’s approach for causal inference needs to bestrengthened. The Agency, in developing its approach, has relied on a number of recentreports that offer models, including the approach taken in the Surgeon General’s Reportsand the conceptual framework set out in the recent Institute of Medicine document,Improving the Presumptive Disability Decision-Making Process for Veterans. Thesedocuments along with the others that are cited do provide useful approaches. TheAgency’s adoption of a five-level set of descriptors of strength of evidence is appropriateand should prove useful for decision-making based on the degree of certainty provided byscientific evidence considered. Through its application, the Agency will be able to refinethe review and synthesis process and, in particular, set precedents for responding toevidence reaching different levels of certainty for causation. For example, wouldevidence reaching the “suggestive” level be a sufficient foundation for altering aNAAQS?There are still limitations of Chapter 1, however. The discussion of association versuscausation is not well formulated and the authors are not clear in discussing key conceptsthat affect interpretation of epidemiological data. For example, confounding and effectmodification are addressed rather superficially, as is measurement error. I urge the staffto continue to refine this material, because it will become a model for future ISAs. Theauthors comment on the complexity of determinants of health and of response to anenvironmental agent, such as sulfur dioxide. However, their approach to doing so islimited and Figure 1-1 is presented with little explanation.37