Robert J. Prill

Wisdom of crowds for robust gene network inference

Reconstructing gene regulatory networks from high-throughput data is a long-standing challenge. Through the Dialogue on Reverse Engineering Assessment and Methods (DREAM) project, we performed a comprehensive blind assessment of over 30 network inference methods on Escherichia coli, Staphylococcus aureus, Saccharomyces cerevisiae and in silico microarray data. We characterize the performance, data requirements and inherent biases of different inference approaches, and we provide guidelines for algorithm application and development. We observed that no single inference method performs optimally across all data sets. In contrast, integration of predictions from multiple inference methods shows robust and high performance across diverse data sets. We thereby constructed high-confidence networks for E. coli and S. aureus, each comprising ∼1,700 transcriptional interactions at a precision of ∼50%. We experimentally tested 53 previously unobserved regulatory interactions in E. coli, of which 23 (43%) were supported. Our results establish community-based methods as a powerful and robust tool for the inference of transcriptional gene regulatory networks.

Revealing strengths and weaknesses of methods for gene network inference

Numerous methods have been developed for inferring gene regulatory networks from expression data, however, both their absolute and comparative performance remain poorly understood. In this paper, we introduce a framework for critical performance assessment of methods for gene network inference. We present an in silico benchmark suite that we provided as a blinded, community-wide challenge within the context of the DREAM (Dialogue on Reverse Engineering Assessment and Methods) project. We assess the performance of 29 gene-network-inference methods, which have been applied independently by participating teams. Performance profiling reveals that current inference methods are affected, to various degrees, by different types of systematic prediction errors. In particular, all but the best-performing method failed to accurately infer multiple regulatory inputs (combinatorial regulation) of genes. The results of this community-wide experiment show that reliable network inference from gene expression data remains an unsolved problem, and they indicate potential ways of network reconstruction improvements.

Towards a Rigorous Assessment of Systems Biology Models: The DREAM3 Challenges

Background Systems biology has embraced computational modeling in response to the quantitative nature and increasing scale of contemporary data sets. The onslaught of data is accelerating as molecular profiling technology evolves. The Dialogue for Reverse Engineering Assessments and Methods (DREAM) is a community effort to catalyze discussion about the design, application, and assessment of systems biology models through annual reverse-engineering challenges. Methodology and Principal Findings We describe our assessments of the four challenges associated with the third DREAM conference which came to be known as the DREAM3 challenges: signaling cascade identification, signaling response prediction, gene expression prediction, and the DREAM3 in silico network challenge. The challenges, based on anonymized data sets, tested participants in network inference and prediction of measurements. Forty teams submitted 413 predicted networks and measurement test sets. Overall, a handful of best-performer teams were identified, while a majority of teams made predictions that were equivalent to random. Counterintuitively, combining the predictions of multiple teams (including the weaker teams) can in some cases improve predictive power beyond that of any single method. Conclusions DREAM provides valuable feedback to practitioners of systems biology modeling. Lessons learned from the predictions of the community provide much-needed context for interpreting claims of efficacy of algorithms described in the scientific literature.

Lessons from the DREAM2 Challenges

Regardless of how creative, innovative, and elegant our computational methods, the ultimate proof of an algorithms worth is the experimentally validated quality of its predictions. Unfortunately, this truism is hard to reduce to practice. Usually, modelers produce hundreds to hundreds of thousands of predictions, most (if not all) of which go untested. In a best‐case scenario, a small subsample of predictions (three to ten usually) is experimentally validated, as a quality control step to attest to the global soundness of the full set of predictions. However, whether this small set is even representative of the global algorithms performance is a question usually left unaddressed. Thus, a clear understanding of the strengths and weaknesses of an algorithm most often remains elusive, especially to the experimental biologists who must decide which tool to use to address a specific problem. In this chapter, we describe the first systematic set of challenges posed to the systems biology community in the framework of the DREAM (Dialogue for Reverse Engineering Assessments and Methods) project. These tests, which came to be known as the DREAM2 challenges, consist of data generously donated by participants to the DREAM project and curated in such a way as to become problems of network reconstruction and whose solutions, the actual networks behind the data, were withheld from the participants. The explanation of the resulting five challenges, a global comparison of the submissions, and a discussion of the best performing strategies are the main topics discussed.

Crowdsourcing Network Inference: The DREAM Predictive Signaling Network Challenge

By aggregating the efforts of the research community, comprehensive and accurate inference of signaling networks may become achievable. Computational analyses of systematic measurements on the states and activities of signaling proteins (as captured by phosphoproteomic data, for example) have the potential to uncover uncharacterized protein-protein interactions and to identify the subset that are important for cellular response to specific biological stimuli. However, inferring mechanistically plausible protein signaling networks (PSNs) from phosphoproteomics data is a difficult task, owing in part to the lack of sufficiently comprehensive experimental measurements, the inherent limitations of network inference algorithms, and a lack of standards for assessing the accuracy of inferred PSNs. A case study in which 12 research groups inferred PSNs from a phosphoproteomics data set demonstrates an assessment of inferred PSNs on the basis of the accuracy of their predictions. The concurrent prediction of the same previously unreported signaling interactions by different participating teams suggests relevant validation experiments and establishes a framework for combining PSNs inferred by multiple research groups into a composite PSN. We conclude that crowdsourcing the construction of PSNs—that is, outsourcing the task to the interested community—may be an effective strategy for network inference.

Geospatial Resolution of Human and Bacterial Diversity with City-Scale Metagenomics

SUMMARY The panoply of microorganisms and other species present in our environment influence human health and disease, especially in cities, but have not been profiled with metagenomics at a city-wide scale. We sequenced DNA from surfaces across the entire New York City (NYC) subway system, the Gowanus Canal, and public parks. Nearly half of the DNA (48%) does not match any known organism; identified organisms spanned 1,688 bacterial, viral, archaeal, and eukaryotic taxa, which were enriched for harmless genera associated with skin (e.g., Acinetobacter). Predicted ancestry of human DNA left on subway surfaces can recapitulate U.S. Census demographic data, and bacterial signatures can reveal a station’s history, such as marine-associated bacteria in a hurricane-flooded station. Some evidence of pathogens was found (Bacillus anthracis), but a lack of reported cases in NYC suggests that the pathogens represent a normal, urban microbiome. This baseline metagenomic map of NYC could help long-term disease surveillance, bioterrorism threat mitigation, and health management in the built environment of cities.

Microbiome changes through ontogeny of a tick pathogen vector

Blacklegged ticks (Ixodes scapularis) are one of the most important pathogen vectors in the United States, responsible for transmitting Lyme disease and other tick‐borne diseases. The structure of a hosts microbial community has the potential to affect the ecology and evolution of the host. We employed high‐throughput sequencing of the 16S rRNA gene V3‐V4 hypervariable regions in the first study to investigate the tick microbiome across all developmental stages (larvae, nymphs, adults). In addition to field‐collected life stages, newly hatched laboratory‐reared larvae were studied to determine the baseline microbial community structure and to assess transovarial transmission. We also targeted midguts and salivary glands due to their importance in pathogen maintenance and transmission. Over 100 000 sequences were produced per life stage replicate. Rickettsia was the most abundant bacterial genus across all sample types matching mostly the Ixodes rickettsial endosymbionts, and its proportion decreased as developmental stage progressed, with the exception of adult females that harboured a mean relative abundance of 97.9%. Laboratory‐reared larvae displayed the lowest bacterial diversity, containing almost exclusively Rickettsia. Many of the remaining bacteria included genera associated with soil, water and plants, suggesting environmental acquisition while off‐host. Female organs exhibited significantly different β‐diversity than the whole tick from which they were derived. Our results demonstrate clear differences in both α‐ and β‐diversity among tick developmental stages and between tick organs and the tick as a whole. Furthermore, field‐acquired bacteria appear to be very important to the overall internal bacterial community of this tick species, with influence from the host bloodmeal appearing limited.

Comprehensive benchmarking and ensemble approaches for metagenomic classifiers

BackgroundOne of the main challenges in metagenomics is the identification of microorganisms in clinical and environmental samples. While an extensive and heterogeneous set of computational tools is available to classify microorganisms using whole-genome shotgun sequencing data, comprehensive comparisons of these methods are limited.ResultsIn this study, we use the largest-to-date set of laboratory-generated and simulated controls across 846 species to evaluate the performance of 11 metagenomic classifiers. Tools were characterized on the basis of their ability to identify taxa at the genus, species, and strain levels, quantify relative abundances of taxa, and classify individual reads to the species level. Strikingly, the number of species identified by the 11 tools can differ by over three orders of magnitude on the same datasets. Various strategies can ameliorate taxonomic misclassification, including abundance filtering, ensemble approaches, and tool intersection. Nevertheless, these strategies were often insufficient to completely eliminate false positives from environmental samples, which are especially important where they concern medically relevant species. Overall, pairing tools with different classification strategies (k-mer, alignment, marker) can combine their respective advantages.ConclusionsThis study provides positive and negative controls, titrated standards, and a guide for selecting tools for metagenomic analyses by comparing ranges of precision, accuracy, and recall. We show that proper experimental design and analysis parameters can reduce false positives, provide greater resolution of species in complex metagenomic samples, and improve the interpretation of results.

DREAMTools: A Python package for scoring collaborative challenges

DREAM challenges are community competitions designed to advance computational methods and address fundamental questions in system biology and translational medicine. Each challenge asks participants to develop and apply computational methods to either predict unobserved outcomes or to identify unknown model parameters given a set of training data. Computational methods are evaluated using an automated scoring metric, scores are posted to a public leaderboard, and methods are published to facilitate community discussions on how to build improved methods. By engaging participants from a wide range of science and engineering backgrounds, DREAM challenges can comparatively evaluate a wide range of statistical, machine learning, and biophysical methods. Here, we describe DREAMTools, a Python package for evaluating DREAM challenge scoring metrics. DREAMTools provides a command line interface that enables researchers to test new methods on past challenges, as well as a framework for scoring new challenges. As of March 2016, DREAMTools includes more than 80% of completed DREAM challenges. DREAMTools complements the data, metadata, and software tools available at the DREAM website http://dreamchallenges.org and on the Synapse platform at https://www.synapse.org. Availability: DREAMTools is a Python package. Releases and documentation are available at http://pypi.python.org/pypi/dreamtools. The source code is available at http://github.com/dreamtools/dreamtools.

Modern Methods for Delineating Metagenomic Complexity

We appreciate the comments of Ackelsberg et al. (Ackelsberg et al., 2015xAckelsberg, J., Rakeman, J., Hughes, S., Peterson, J., Mead, P., Schriefer, M., Kingry, L., Hoffmaster, A., and Gee, J. Cell Syst. 2015; 1: 4–5Abstract | Full Text | Full Text PDF | Scopus (1)See all ReferencesAckelsberg et al., 2015) and have decided to revise the paper (Afshinnekoo et al., 2015xAfshinnekoo, E., Meydan, C., Chowdhury, S., Jaroudi, D., Boyer, C., Bernstein, N., Maritz, J.M., Reeves, D., Gandara, J., Chhangawala, S. et al. Cell Syst. 2015; 1: 72–87Abstract | Full Text | Full Text PDF | Scopus (23)See all ReferencesAfshinnekoo et al., 2015) as follows:Figure 3B has been corrected to show the general coverage of the Yersinia pestis pMT1 plasmid, but not the murine toxin gene (yMT). The initial claim of “…consistent 20× coverage across the murine toxin gene…” was erroneously based on looking at annotations from related plasmids and comparing different reference sequences. In actuality no reads mapped to the yMT gene.The result of low coverage to the Bacillus anthracis plasmids (pXO1 and pXO2) and no evidence of plcR SNP—an often defining feature of anthrax—is now reported in the Results section.The language in the Summary, Results, and Discussion has been revised, and speculative text about pathogenic organisms has been deleted. We now state that although all our metagenomic analysis tools identified reads with similarity to B. anthracis and Y. pestis sequences, there is minimal coverage to the backbone genome of these organisms, and there is no strong evidence to suggest these organisms are in fact present and no evidence of pathogenicity.Furthermore, in regards to the concerns of the culture methods we have posted subsequent details on the study website (http://www.pathomap.org/2015/04/13/culture-methods/) and below.A second culture experiment was performed to address the question of antibiotic resistance (Afshinnekoo et al., 2015xAfshinnekoo, E., Meydan, C., Chowdhury, S., Jaroudi, D., Boyer, C., Bernstein, N., Maritz, J.M., Reeves, D., Gandara, J., Chhangawala, S. et al. Cell Syst. 2015; 1: 72–87Abstract | Full Text | Full Text PDF | Scopus (23)See all ReferencesAfshinnekoo et al., 2015, Figure 4A). Bacteria were cultured in LB agar and then spread onto LB plates, after lawn growth, single colonies were picked and then plated onto antibiotic plates (kanamycin – 50 ug/ml, chloramphenicol – 35 ug/ml, and ampicillin – 100 ug/ml) and growth was assessed. Plates were incubated at 37°C. As a control, air samples were taken and cultured at every location. In all cases, these did not yield growth. The non-selective plate done last when replica plating also serves as a control. There was no quantitative confirmation of bacterial versus non-bacterial organisms, although there was no observable fungal growth in the samples. Further experiments are being done to dive deeper into the question of viability of microorganisms on the subway system as well as the presence of antibiotic-resistant bacteria.The field of metagenomics is relatively new but has great potential to serve an incredibly important role both in our understanding of the world around us—with key applications in the built environment—as well as the clinical realm. Nevertheless, there are still major hurdles and challenges that the field faces in order to realize this potential. We welcome and appreciate the discussion (http://microbe.net/2015/02/17/the-long-road-from-data-to-wisdom-and-from-dna-to-pathogen/) prompted by our study, and we anticipate that this large dataset will enable further experimentation, additional testing of taxonomic tools, and hopefully help in developing methodologies for metagenomic analysis.