Mary S. Wolfe

Systematic Review and Evidence Integration for Literature-Based Environmental Health Science Assessments

Background: Systematic-review methodologies provide objectivity and transparency to the process of collecting and synthesizing scientific evidence in reaching conclusions on specific research questions. There is increasing interest in applying these procedures to address environmental health questions. Objectives: The goal was to develop a systematic-review framework to address environmental health questions by extending approaches developed for clinical medicine to handle the breadth of data relevant to environmental health sciences (e.g., human, animal, and mechanistic studies). Methods: The Office of Health Assessment and Translation (OHAT) adapted guidance from authorities on systematic-review and sought advice during development of the OHAT Approach through consultation with technical experts in systematic review and human health assessments, as well as scientific advisory groups and the public. The method was refined by considering expert and public comments and through application to case studies. Results and Discussion: Here we present a seven-step framework for systematic review and evidence integration for reaching hazard identification conclusions: 1) problem formulation and protocol development, 2) search for and select studies for inclusion, 3) extract data from studies, 4) assess the quality or risk of bias of individual studies, 5) rate the confidence in the body of evidence, 6) translate the confidence ratings into levels of evidence, and 7) integrate the information from different evidence streams (human, animal, and “other relevant data” including mechanistic or in vitro studies) to develop hazard identification conclusions. Conclusion: The principles of systematic review can be successfully applied to environmental health questions to provide greater objectivity and transparency to the process of developing conclusions. Citation: Rooney AA, Boyles AL, Wolfe MS, Bucher JR, Thayer KA. 2014. Systematic review and evidence integration for literature-based environmental health science assessments. Environ Health Perspect 122:711–718; http://dx.doi.org/10.1289/ehp.1307972

GRADE: Assessing the quality of evidence in environmental and occupational health

There is high demand in environmental health for adoption of a structured process that evaluates and integrates evidence while making decisions and recommendations transparent. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) framework holds promise to address this demand. For over a decade, GRADE has been applied successfully to areas of clinical medicine, public health, and health policy, but experience with GRADE in environmental and occupational health is just beginning. Environmental and occupational health questions focus on understanding whether an exposure is a potential health hazard or risk, assessing the exposure to understand the extent and magnitude of risk, and exploring interventions to mitigate exposure or risk. Although GRADE offers many advantages, including its flexibility and methodological rigor, there are features of the different sources of evidence used in environmental and occupational health that will require further consideration to assess the need for method refinement. An issue that requires particular attention is the evaluation and integration of evidence from human, animal, in vitro, and in silico (computer modeling) studies when determining whether an environmental factor represents a potential health hazard or risk. Assessment of the hazard of exposures can produce analyses for use in the GRADE evidence-to-decision (EtD) framework to inform risk-management decisions about removing harmful exposures or mitigating risks. The EtD framework allows for grading the strength of the recommendations based on judgments of the certainty in the evidence (also known as quality of the evidence), as well as other factors that inform recommendations such as social values and preferences, resource implications, and benefits. GRADE represents an untapped opportunity for environmental and occupational health to make evidence-based recommendations in a systematic and transparent manner. The objectives of this article are to provide an overview of GRADE, discuss GRADEs applicability to environmental health, and identify priority areas for method assessment and development.

Report of Partial findings from the National Toxicology Program Carcinogenesis Studies of Cell Phone Radiofrequency Radiation in Hsd: Sprague Dawley® SD rats (Whole Body Exposure)

The U.S. National Toxicology Program (NTP) has carried out extensive rodent toxicology and carcinogenesis studies of radiofrequency radiation (RFR) at frequencies and modulations used in the U.S. telecommunications industry. This report presents partial findings from these studies. The occurrences of two tumor types in male Harlan Sprague Dawley rats exposed to RFR, malignant gliomas in the brain and schwannomas of the heart, were considered of particular interest and are the subject of this report. The findings in this report were reviewed by expert peer reviewers selected by the NTP and National Institutes of Health (NIH). These reviews and responses to comments are included as appendices to this report, and revisions to the current document have incorporated and addressed these comments. When the studies are completed, they will undergo additional peer review before publication in full as part of the NTPs Toxicology and Carcinogenesis Technical Reports Series. No portion of this work has been submitted for publication in a scientific journal. Supplemental information in the form of four additional manuscripts has or will soon be submitted for publication. These manuscripts describe in detail the designs and performance of the RFR exposure system, the dosimetry of RFR exposures in rats and mice, the results to a series of pilot studies establishing the ability of the animals to thermoregulate during RFR exposures, and studies of DNA damage. (1) Capstick M, Kuster N, Kuhn S, Berdinas-Torres V, Wilson P, Ladbury J, Koepke G, McCormick D, Gauger J, and Melnick R. A radio frequency radiation reverberation chamber exposure system for rodents; (2) Yijian G, Capstick M, McCormick D, Gauger J, Horn T, Wilson P, Melnick RL, and Kuster N. Life time dosimetric assessment for mice and rats exposed to cell phone radiation; (3) Wyde ME, Horn TL, Capstick M, Ladbury J, Koepke G, Wilson P, Stout MD, Kuster N, Melnick R, Bucher JR, and McCormick D. Pilot studies of the National Toxicology Program’s cell phone radiofrequency radiation reverberation chamber exposure system; (4) Smith-Roe SL, Wyde ME, Stout MD, Winters J, Hobbs CA, Shepard KG, Green A, Kissling GE, Tice RR, Bucher JR, and Witt KL. Evaluation of the genotoxicity of cell phone radiofrequency radiation in male and female rats and mice following subchronic exposure. SUMMARY The purpose of this communication is to report partial findings from a series of radiofrequency radiation (RFR) cancer studies in rats performed under the auspices of the U.S. National Toxicology Program (NTP).1 This report contains peer-reviewed, neoplastic and hyperplastic findings only in the brain and heart of Hsd:Sprague Dawley® SD® (HSD) rats exposed to RFR starting in utero and continuing throughout their lifetimes. These studies found low incidences of malignant gliomas in the brain and schwannomas in the heart of male rats exposed to RFR of the two types [Code Division Multiple Access (CDMA) and Global System for Mobile Communications (GSM)] currently used in U.S. wireless networks. Potentially preneoplastic lesions were also observed in the brain and heart of male rats exposed to RFR. The review of partial study data in this report has been prompted by several factors. Given the widespread global usage of mobile communications among users of all ages, even a very small increase in the incidence of disease resulting from exposure to RFR could have broad implications for public health. There is a high level of public and media interest regarding the safety of cell phone RFR and the specific results of these NTP studies. Lastly, the tumors in the brain and heart observed at low incidence in male rats exposed to GSM-and CDMA-modulated cell phone RFR in this study are of a type similar to tumors observed in some epidemiology studies of cell phone use. These findings appear to support the International Agency for Research on Cancer (IARC) conclusions regarding the possible carcinogenic potential of RFR.2 It is important to note that this document reviews only the findings from the brain and heart and is not a complete report of all findings from the NTP’s studies. Additional data from these studies in Hsd:Sprague Dawley® SD® (Harlan) rats and similar studies conducted in B6C3F1/N mice are currently under evaluation and will be reported together with the current findings in two forthcoming NTP Technical Reports.

Toxicity characterization of environmental chemicals by the US National Toxicology Program: an overview

The US National Toxicology Program (NTP) is an interagency program whose mission is to evaluate agents of public health concern by developing and applying the tools of modern toxicology and molecular biology. Chemicals substances or physical agents selected for toxicology and carcinogenesis evaluations by the NTP are usually studied in a series of subacute (14-day exposure), subchronic (90-day exposure) and chronic (2-year exposure) studies in rodents. The NTP has published more than 500 reports of the findings and conclusions from its toxicology and carcinogenesis studies. In more specialized studies, the NTP also evaluates adverse effects on the structure and function of the immune, reproductive, nervous, and respiratory systems. The program attempts to evaluate and appropriately incorporate new technologies to improve the way we study the toxicity of chemicals. For example, the program has extensively evaluated several transgenic mouse models for their potential use as short-term cancer screens and has been a full participant in an international effort to examine their usefulness in pharmaceutical registration. Toxicogenomics, an emerging scientific field that examines the expression of thousands of genes simultaneously in response to chemical exposure, holds promise for future application to better understand the underlying mechanisms of chemical toxicity. A number of public health issues being addressed by the NTP are not only of national importance but also have global impact, such as the potential for endocrine disruptors to influence development and carcinogenesis and the safety of herbal medicines and dietary supplements. The program participates in the preparation of national and international toxicity testing guidelines and the findings from NTP studies are widely used for risk assessments by international organizations and federal agencies. The NTP maintains databases that contain toxicity, and health and safety information on a large number of chemicals. These databases are available from the NTP web site (http://ntp-server.niehs.nih.gov) and are accessed over 100000 times a month from around the world.

Implementing Systematic Review at the National Toxicology Program: Status and Next Steps

The National Toxicology Program (NTP), an interagency program headquartered at the National Institute of Environmental Health Sciences (NIEHS), carries out a broad range of toxicology research and testing and serves as a resource for identification of substances in our environment that are hazards for human health. One of the ways that the NTP identifies hazards is through carrying out literature-based health assessments. Approximately 2 years ago we began exploring systematic-review methodology as a means to enhance transparency and increase efficiency in summarizing and synthesizing findings from studies in our literature-based health assessments. A systematic review uses an explicit, prespecified approach to identify, select, assess, and appraise the data from studies that focus on addressing a specific scientific question (Institute of Medicine 2011). Although traditionally used to grade the quality of evidence and strength of scientific support for recommendations for clinical practice guidelines and healthcare interventions [Agency for Healthcare Research and Quality (AHRQ) 2012; Guyatt et al. 2011; Higgins and Green 2011], we—and others—were interested in how systematic review methodology might be applied to environmental health questions (Agency for Toxic Substances and Disease Registry 2012; National Research Council 2011; Silbergeld and Scherer 2013; U.S. Environmental Protection Agency 2013; Woodruff and Sutton 2011). With the establishment of the Office of Health Assessment and Translation (OHAT) in 2011, the NIEHS launched a new problem-solving resource for the NTP, particularly with respect to identification of noncancer hazards in our environment (Bucher et al. 2011). OHAT took the lead in investigating how systematic review methodology might be used by the NTP. We embraced systematic review methodology as a useful approach for providing thorough documentation of the steps, inputs, and decisions in a literature-based evaluation. However, we also recognized the necessity to extend existing systematic review methods to accommodate our need in environmental health to integrate data from multiple evidence streams (human, animal, in vitro) and focus on observational human studies rather than on the randomized clinical trials more commonly encountered in the field of health-care intervention (NTP 2012a, 2012b). In late February 2013, the NTP released the Draft OHAT Approach for Systematic Review and Evidence Integration for Literature-based Health Assessments – February 2013 [Draft OHAT Approach; Department of Health and Human Services (DHHS) 2013] for public comment; the deadline for receipt of comments is 11 June 2013. The Draft OHAT Approach adopts or adapts guidance from authoritative systematic review groups (AHRQ 2012; Guyatt et al. 2011; Higgins and Green 2011) to handle the breadth of data from human, animal, in vitro, and mechanistic studies relevant for addressing environmental health questions. In developing a draft approach, OHAT sought advice on systematic review through educational webinars and consultation with technical experts, the NTP Executive Committee, a working group of the NTP Board of Scientific Counselors, the NTP Board of Scientific Counselors, and the public. The draft approach involves a seven-step framework for incorporating systematic review methodology into OHAT literature-based health assessments. In early April of 2013, OHAT will release protocols for two case studies to illustrate application of this framework in specific evaluations. We will test our approach in these case studies to help determine whether additional refinement or revision to the Draft OHAT Approach might be needed. To help the public understand the draft approach and protocols, the NTP will hold a web-based informational meeting on 23 April 2013 to provide an overview of the framework, describe the contents of the case-study protocols, and respond to questions (DHHS 2013). Our intent is to carefully consider all public comments received on the draft approach and to present the Draft OHAT Approach to the NTP Board of Scientific Counselors at its meeting on 25–26 June 2013, with discussion by the NTP of any plans to update the document on the basis of the public’s input. Moving forward, our goal is to increase efficiency and provide greater transparency to the rigorous and objective approach that has been the hallmark of OHAT literature-based health assessments.

Evaluating the Impact of the U.S. National Toxicology Program: A Case Study on Hexavalent Chromium.

Background: Evaluating the impact of federally funded research with a broad, methodical, and objective approach is important to ensure that public funds advance the mission of federal agencies. Objectives: We aimed to develop a methodical approach that would yield a broad assessment of National Toxicology Program’s (NTP’s) effectiveness across multiple sectors and demonstrate the utility of the approach through a case study. Methods: A conceptual model was developed with defined activities, outputs (products), and outcomes (proximal, intermediate, distal) and applied retrospectively to NTP’s research on hexavalent chromium (CrVI). Proximal outcomes were measured by counting views of and requests for NTP’s products by external stakeholders. Intermediate outcomes were measured by bibliometric analysis. Distal outcomes were assessed through Web and LexisNexis searches for documents related to legislation or regulation changes. Results: The approach identified awareness of NTP’s work on CrVI by external stakeholders (proximal outcome) and citations of NTP’s research in scientific publications, reports, congressional testimonies, and legal and policy documents (intermediate outcome). NTP’s research was key to the nation’s first-ever drinking water standard for CrVI adopted by California in 2014 (distal outcome). By applying this approach to a case study, the utility and limitations of the approach were identified, including challenges to evaluating the outcomes of a research program. Conclusions: This study identified a broad and objective approach for assessing NTP’s effectiveness, including methodological needs for more thorough and efficient impact assessments in the future. Citation: Xie Y, Holmgren S, Andrews DMK, Wolfe MS. 2017. Evaluating the impact of the U.S. National Toxicology Program: a case study on hexavalent chromium. Environ Health Perspect 125:181–188; http://dx.doi.org/10.1289/EHP21

Evaluation of a rule-based method for epidemiological document classification towards the automation of systematic reviews

INTRODUCTION Most data extraction efforts in epidemiology are focused on obtaining targeted information from clinical trials. In contrast, limited research has been conducted on the identification of information from observational studies, a major source for human evidence in many fields, including environmental health. The recognition of key epidemiological information (e.g., exposures) through text mining techniques can assist in the automation of systematic reviews and other evidence summaries. METHOD We designed and applied a knowledge-driven, rule-based approach to identify targeted information (study design, participant population, exposure, outcome, confounding factors, and the country where the study was conducted) from abstracts of epidemiological studies included in several systematic reviews of environmental health exposures. The rules were based on common syntactical patterns observed in text and are thus not specific to any systematic review. To validate the general applicability of our approach, we compared the data extracted using our approach versus hand curation for 35 epidemiological study abstracts manually selected for inclusion in two systematic reviews. RESULTS The returned F-score, precision, and recall ranged from 70% to 98%, 81% to 100%, and 54% to 97%, respectively. The highest precision was observed for exposure, outcome and population (100%) while recall was best for exposure and study design with 97% and 89%, respectively. The lowest recall was observed for the population (54%), which also had the lowest F-score (70%). CONCLUSION The generated performance of our text-mining approach demonstrated encouraging results for the identification of targeted information from observational epidemiological study abstracts related to environmental exposures. We have demonstrated that rules based on generic syntactic patterns in one corpus can be applied to other observational study design by simple interchanging the dictionaries aiming to identify certain characteristics (i.e., outcomes, exposures). At the document level, the recognised information can assist in the selection and categorization of studies included in a systematic review.

Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR)

The second meeting of the International Collaboration for Automation of Systematic Reviews (ICASR) was held 3–4 October 2016 in Philadelphia, Pennsylvania, USA. ICASR is an interdisciplinary group whose aim is to maximize the use of technology for conducting rapid, accurate, and efficient systematic reviews of scientific evidence. Having automated tools for systematic review should enable more transparent and timely review, maximizing the potential for identifying and translating research findings to practical application. The meeting brought together multiple stakeholder groups including users of summarized research, methodologists who explore production processes and systematic review quality, and technologists such as software developers, statisticians, and vendors. This diversity of participants was intended to ensure effective communication with numerous stakeholders about progress toward automation of systematic reviews and stimulate discussion about potential solutions to identified challenges. The meeting highlighted challenges, both simple and complex, and raised awareness among participants about ongoing efforts by various stakeholders. An outcome of this forum was to identify several short-term projects that participants felt would advance the automation of tasks in the systematic review workflow including (1) fostering better understanding about available tools, (2) developing validated datasets for testing new tools, (3) determining a standard method to facilitate interoperability of tools such as through an application programming interface or API, and (4) establishing criteria to evaluate the quality of tools’ output. ICASR 2016 provided a beneficial forum to foster focused discussion about tool development and resources and reconfirm ICASR members’ commitment toward systematic reviews’ automation.