Douglas A. Popken

Developing scientific confidence in HTS-derived prediction models: lessons learned from an endocrine case study.

High throughput (HTS) and high content (HCS) screening methods show great promise in changing how hazard and risk assessments are undertaken, but scientific confidence in such methods and associated prediction models needs to be established prior to regulatory use. Using a case study of HTS-derived models for predicting in vivo androgen (A), estrogen (E), thyroid (T) and steroidogenesis (S) endpoints in endocrine screening assays, we compare classification (fitting) models to cross validation (prediction) models. The more robust cross validation models (based on a set of endocrine ToxCast™ assays and guideline in vivo endocrine screening studies) have balanced accuracies from 79% to 85% for A and E, but only 23% to 50% for T and S. Thus, for E and A, HTS results appear promising for initial use in setting priorities for endocrine screening. However, continued research is needed to expand the domain of applicability and to develop more robust HTS/HCS-based prediction models prior to their use in other regulatory applications. Based on the lessons learned, we propose a framework for documenting scientific confidence in HTS assays and the prediction models derived therefrom. The documentation, transparency and the scientific rigor involved in addressing the elements in the proposed Scientific Confidence Framework could aid in discussions and decisions about the prediction accuracy needed for different applications.

Human Health Risk Assessment of Penicillin/ Aminopenicillin Resistance in Enterococci Due to Penicillin Use in Food Animals

Penicillin and ampicillin drugs are approved for use in food animals in the United States to treat, control, and prevent diseases, and penicillin is approved for use to improve growth rates in pigs and poultry. This article considers the possibility that such uses might increase the incidence of ampicillin-resistant Enterococcus faecium (AREF) of animal origin in human infections, leading to increased hospitalization and mortality due to reduced response to ampicillin or penicillin. We assess the risks from continued use of penicillin-based drugs in food animals in the United States, using several assumptions to overcome current scientific uncertainties and data gaps. Multiplying the total at-risk population of intensive care unit (ICU) patients by a series of estimated factors suggests that not more than 0.04 excess mortalities per year (under conservative assumptions) to 0.14 excess mortalities per year (under very conservative assumptions) might be prevented in the whole U.S. population if current use of penicillin drugs in food animals were discontinued and if this successfully reduced the prevalence of AREF infections among ICU patients. These calculations suggest that current penicillin usage in food animals in the United States presents very low (possibly zero) human health risks.

Causal versus spurious spatial exposure–response associations in health risk analysis

Abstract Many recent health risk assessments have noted that adverse health outcomes are significantly statistically associated with proximity to suspected sources of health hazard, such as manufacturing plants or point sources of air pollution. Using geographic proximity to sources as surrogates for exposure to (possibly unknown) releases, spatial ecological studies have identified potential adverse health effects based on significant regression coefficients between risk rates and distances from sources in multivariate statistical risk models. Although this procedure has been fruitful in identifying exposure–response associations, it is not always clear whether the resulting regression coefficients have valid causal interpretations. Spurious spatial regression and other threats to valid causal inference may undermine practical efforts to causally link health effects to geographic sources, even when there are clear statistical associations between them. This paper demonstrates the methodological problems by examining statistical associations and regression coefficients between spatially distributed exposure and response variables in a realistic data set for California. We find that distance from “nonsense” sources (such as arbitrary points or lines) are highly statistically significant predictors of cause-specific risks, such as traffic fatalities and incidence of Kaposi’s sarcoma. However, the signs of such associations typically depend on the distance scale chosen. This is consistent with theoretical analyses showing that random spatial trends (which tend to fluctuate in sign), rather than true causal relations, can create statistically significant regression coefficients: spatial location itself becomes a confounder for spatially distributed exposure and response variables. Hence, extreme caution and careful application of spatial statistical methods are warranted before interpreting proximity-based exposure–response relations as evidence of a possible or probable causal relation.

Optimal Tracking and Testing of U.S. and Canadian Herds for BSE: A Value-of-Information (VOI) Approach

The U.S. Department of Agriculture (USDA) tests a subset of cattle slaughtered in the United States for bovine spongiform encephalitis (BSE). Knowing the origin of cattle (U.S. vs. Canadian) at testing could enable new testing or surveillance policies based on the origin of cattle testing positive. For example, if a Canadian cow tests positive for BSE, while no U.S. origin cattle do, the United States could subject Canadian cattle to more stringent testing. This article illustrates the application of a value-of-information (VOI) framework to quantify and compare potential economic costs to the United States of implementing tracking cattle origins to the costs of not doing so. The potential economic value of information from a tracking program is estimated to exceed its costs by more than five-fold if such information can reduce future losses in export and domestic markets and reduce future testing costs required to reassure or win back customers. Sensitivity analyses indicate that this conclusion is somewhat robust to many technical, scientific, and market uncertainties, including the current prevalence of BSE in the United States and/or Canada and the likely reactions of consumers to possible future discoveries of BSE in the United States and/or Canada. Indeed, the potential value of tracking information is great enough to justify locating and tracking Canadian cattle already in the United States when this can be done for a reasonable cost. If aggressive tracking and testing can win back lost exports, then the VOI of a tracking program may increase to over half a billion dollars per year.

Overcoming Confirmation Bias in Causal Attribution: A Case Study of Antibiotic Resistance Risks

When they do not use formal quantitative risk assessment methods, many scientists (like other people) make mistakes and exhibit biases in reasoning about causation, if-then relations, and evidence. Decision-related conclusions or causal explanations are reached prematurely based on narrative plausibility rather than adequate factual evidence. Then, confirming evidence is sought and emphasized, but disconfirming evidence is ignored or discounted. This tendency has serious implications for health-related public policy discussions and decisions. We provide examples occurring in antimicrobial health risk assessments, including a case study of a recently reported positive relation between virginiamycin (VM) use in poultry and risk of resistance to VM-like (streptogramin) antibiotics in humans. This finding has been used to argue that poultry consumption causes increased resistance risks, that serious health impacts may result, and therefore use of VM in poultry should be restricted. However, the original study compared healthy vegetarians to hospitalized poultry consumers. Our examination of the same data using conditional independence tests for potential causality reveals that poultry consumption acted as a surrogate for hospitalization in this study. After accounting for current hospitalization status, no evidence remains supporting a causal relationship between poultry consumption and increased streptogramin resistance. This example emphasizes both the importance and the practical possibility of analyzing and presenting quantitative risk information using data analysis techniques (such as Bayesian model averaging (BMA) and conditional independence tests) that are as free as possible from potential selection, confirmation, and modeling biases.

Quantitative Assessment of Human MRSA Risks from Swine

The public health community, news media, and members of the general public have expressed significant concern that methicillin-resistant Staphylococcus aureus (MRSA) transmitted from pigs to humans may harm human health. Studies of the prevalence and dynamics of swine-associated (ST398) MRSA have sampled MRSA at discrete points in the presumed causative chain leading from swine to human patients, including sampling bacteria from live pigs, retail meats, farm workers, and hospital patients. Nonzero prevalence is generally interpreted as indicating a potential human health hazard from MRSA infections, but quantitative assessments of resulting risks are not usually provided. This article integrates available data from several sources to construct a conservative (plausible upper bound) probability estimate for the actual human health harm (MRSA infections and fatalities) arising from ST398-MRSA from pigs. The model provides plausible upper bounds of approximately one excess human infection per year among all U.S. pig farm workers, and one human infection per 31 years among the remaining total population of the United States. These results assume the possibility of transmission events not yet observed, so additional data collection may reduce these estimates further.

A cautionary tale: The characteristics of two-dimensional distributions and their effects on epidemiological studies employing an ecological design

Abstract In recent years, many spatial epidemiological studies that use proximity of subjects to putative sources as a surrogate for exposure have been published and are increasingly cited as evidence of environmental problems requiring public health interventions. In these studies, the simple finding of a significant, positive association between proximity and disease incidence has been interpreted as evidence of causality. However, numerous authors have pointed out limitations to such interpretations. This, the first of two companion studies, examines the effects of analyzing (real and simulated) spatial data using logistic regression. Simulation is also employed to explore the statistical power of such analyses to detect true effects, quantify the probabilities of Type I and Type II errors, and to evaluate a proposed mechanism that explains the observed effects. Results indicate that, even when the odds ratios of cases and controls are regressed against random or nonsense sources, significant, positive associations are observed at frequencies substantially greater than chance. These frequencies increase when targets are highly non-uniformly distributed such that, for example, false-positive associations are more likely than not when odds ratios are regressed against the actual distribution of ultramafic rocks in California. The coefficients of true, causal associations are substantially attenuated under realistic conditions so that, absent corroborating analyses, there is no non-arbitrary means of distinguishing causal from spurious or real but non-causal associations. Factors affecting where people choose to live act as powerful confounders, creating spurious or real but non-causal associations between exposure and response variables (as well as between other pairs of variables). Consequently, future epidemiological studies that use proximity as a surrogate for exposure should be required to include adequate negative control analyses and/or other kinds of corroborating analyses before they are accepted for publication.

Quantifying Human Health Risks from Animal Antimicrobials

In 1969, the Joint Committee on the Use of Antibiotics in Animal Husbandry and Veterinary Medicine in the United Kingdom warned that uncontrolled use of similar antimicrobials in humans and food animals might promote the emergence of resistant strains of foodborne bacteria that could endanger human health and compromise the effectiveness of antimicrobial therapies in human patients (Swann 1969). The Animal Health Institute (AHI) and its member companies collaborated with Cox Associates, an operations research consulting company, to develop and apply new, practical, quantitative risk assessment (QRA) modeling methods to assess the previously impossible-to-quantify risks (and benefits) to human health from continued use of animal antimicrobials. We came to some surprising conclusions that were robust to many uncertainties. Among these were that antimicrobials that benefit animal health may benefit human health, while regulatory interventions that seek to reduce antimicrobial resistance in animals may unintentionally increase illness rates (and hence antimicrobial use and resistance rates) in humans. These new QRA models and methods enable industry and regulatory decision makers to quantify and compare the probable human health consequences of alternative animal antimicrobial use plans and to design more effective approaches to protect human and animal health.

Model identification and optimization for operational simulation

This paper describes initial research to define and demonstrate an integrated set of algorithms for conducting high-level Operational Simulations. In practice, an Operational Simulation would be used during an ongoing military mission to monitor operations, update state information, compare actual versus planned states, and suggest revised alternative Courses of Action. Significant technical challenges to this realization result from the size and complexity of the problem domain, the inherent uncertainty of situation assessments, and the need for immediate answers. Taking a top-down approach, we initially define the problem with respect to high-level military planning. By narrowing the state space we are better able to focus on model, data, and algorithm integration issues without getting sidetracked by issues specific to any single application or implementation. We propose three main functions in the planning cycle: situation assessment, parameter update, and plan assessment and prediction. Situation assessment uses hierarchical Bayes Networks to estimate initial state probabilities. A parameter update function based on Hidden Markov Models then produces revised state probabilities and state transition probabilities - model identification. Finally, the plan assessment and prediction function uses these revised estimates for simulation-based prediction as well as for determining optimal policies via Markov Decision Processes and simulation-optimization heuristics.

Quantifying Human Health Risks Caused by Toxoplasmosis from Open System Production of Swine

ABSTRACT Open livestock production systems, including free-range and organic livestock systems, seek to improve the welfare of animals by letting them roam in unconfined spaces. This increases their exposure to potentially harmful micro-organisms. For example, swine in open production systems have a much greater risk of Toxoplasma gondii infection. When transmitted through the food chain, T. gondii threatens human health, especially in unborn children of women infected during pregnancy, as well as the lives of patients with compromised immune systems. By contrast, conventional total confinement production systems can now keep this human health risk at or near zero. This article describes a probabilistic risk simulation model that quantified the tradeoff between greater use of open swine production systems and increased cases of toxoplasmosis in humans. The model predicts that every 1804 pigs shifted from conventional total confinement to open production (95% confidence interval 747–9520) would cause the loss of one additional human quality-adjusted life year (QALY), and that increasing the fraction of U.S. swine raised in open/free range operations by 0.1% (approx. 65,000 pigs) would cause a loss of approximately 36 human QALYs per year, including between 1 and 2 extra adult deaths per year.