Narayan Desai

Unlocking the potential of metagenomics through replicated experimental design

Metagenomics holds enormous promise for discovering novel enzymes and organisms that are biomarkers or drivers of processes relevant to disease, industry and the environment. In the past two years, we have seen a paradigm shift in metagenomics to the application of cross-sectional and longitudinal studies enabled by advances in DNA sequencing and high-performance computing. These technologies now make it possible to broadly assess microbial diversity and function, allowing systematic investigation of the largely unexplored frontier of microbial life. To achieve this aim, the global scientific community must collaborate and agree upon common objectives and data standards to enable comparative research across the Earths microbiome. Improvements in comparability of data will facilitate the study of biotechnologically relevant processes, such as bioprospecting for new glycoside hydrolases or identifying novel energy sources.

Meeting Report: The Terabase Metagenomics Workshop and the Vision of an Earth Microbiome Project

Between July 18th and 24th 2010, 26 leading microbial ecology, computation, bioinformatics and statistics researchers came together in Snowbird, Utah (USA) to discuss the challenge of how to best characterize the microbial world using next-generation sequencing technologies. The meeting was entitled “Terabase Metagenomics” and was sponsored by the Institute for Computing in Science (ICiS) summer 2010 workshop program. The aim of the workshop was to explore the fundamental questions relating to microbial ecology that could be addressed using advances in sequencing potential. Technological advances in next-generation sequencing platforms such as the Illumina HiSeq 2000 can generate in excess of 250 billion base pairs of genetic information in 8 days. Thus, the generation of a trillion base pairs of genetic information is becoming a routine matter. The main outcome from this meeting was the birth of a concept and practical approach to exploring microbial life on earth, the Earth Microbiome Project (EMP). Here we briefly describe the highlights of this meeting and provide an overview of the EMP concept and how it can be applied to exploration of the microbiome of each ecosystem on this planet.

From genomics to metagenomics.

Next-generation sequencing has changed metagenomics. However, sequencing DNA is no longer the bottleneck, rather, the bottleneck is computational analysis and also interpretation. Computational cost is the obvious issue, as is tool limitations, considering most of the tools we routinely use have been built for clonal genomics or are being adapted to microbial communities. The current trend in metagenomics analysis is toward reducing computational costs through improved algorithms and through analysis strategies. Data sharing and interoperability between tools are critical, since computation for metagenomic datasets is very high.

Magellan: experiences from a science cloud

Cloud resources promise to be an avenue to address new categories of scientific applications including data-intensive science applications, on-demand/surge computing, and applications that require customized software environments. However, there is a limited understanding on how to operate and use clouds for scientific applications. Magellan, a project funded through the Department of Energys (DOE) Advanced Scientific Computing Research (ASCR) program, is investigating the use of cloud computing for science at the Argonne Leadership Computing Facility (ALCF) and the National Energy Research Scientific Computing Facility (NERSC). In this paper, we detail the experiences to date at both sites and identify the gaps and open challenges from both a resource provider as well as application perspective.

Using clouds for metagenomics: A case study

Cutting-edge sequencing systems produce data at a prodigious rate; and the analysis of these datasets requires significant computing resources. Cloud computing provides a tantalizing possibility for on-demand access to computing resources. However, many open questions remain. We present here a performance assessment of BLAST on real metagenomics data in a cloud setting in order to determine the viability of this approach. BLAST is one of the premier applications in bioinformatics and computational biology and is assumed to consume the vast majority of resources in that area.

Analyzing and adjusting user runtime estimates to improve job scheduling on the Blue Gene/P

Backfilling and short-job-first are widely acknowledged enhancements to the simple but popular first-come, first-served job scheduling policy. However, both enhancements depend on user-provided estimates of job runtime, which research has repeatedly shown to be inaccurate. We have investigated the effects of this inaccuracy on backfilling and different queue prioritization policies, determining which part of the scheduling policy is most sensitive. Using these results, we have designed and implemented several estimation-adjusting schemes based on historical data. We have evaluated these schemes using workload traces from the Blue Gene/P system at Argonne National Laboratory. Our experimental results demonstrate that dynamically adjusting job runtime estimates can improve job scheduling performance by up to 20%.

Fault-aware, utility-based job scheduling on Blue, Gene/P systems

Job scheduling on large-scale systems is an increasingly complicated affair, with numerous factors influencing scheduling policy. Addressing these concerns results in sophisticated scheduling policies that can be difficult to reason about. In this paper, we present a general utility-based scheduling framework to balance various scheduling requirements and priorities. It enables system owners to customize scheduling policies under different circumstances without changing the scheduling code. We also develop a fault-aware job allocation strategy for Blue Gene/P systems to address the increasing concern of system failures. We demonstrate the effectiveness of these facilities by means of event-driven simulations with real job traces collected from the production Blue Gene/P system at Argonne National Laboratory.

Reducing Energy Costs for IBM Blue Gene/P via Power-Aware Job Scheduling

Energy expense is becoming increasingly dominant in the operating costs of high-performance computing (HPC) systems. At the same time, electricity prices vary significantly at different times of the day. Furthermore, job power profiles also differ greatly, especially on HPC systems. In this paper, we propose a smart, power-aware job scheduling approach for HPC systems based on variable energy prices and job power profiles. In particular, we propose a 0-1 knapsack model and demonstrate its flexibility and effectiveness for scheduling jobs, with the goal of reducing energy cost and not degrading system utilization. We design scheduling strategies for Blue Gene/P, a typical partition-based system. Experiments with both synthetic data and real job traces from production systems show that our power-aware job scheduling approach can reduce the energy cost significantly, up to 25 %, with only slight impact on system utilization.

The complete genome sequence for putative H₂- and S-oxidizer Candidatus Sulfuricurvum sp., assembled de novo from an aquifer-derived metagenome.

We reconstructed the complete 2.4 Mb-long genome of a previously uncultivated epsilonproteobacterium, Candidatus Sulfuricurvum sp. RIFRC-1, via assembly of short-read shotgun metagenomic data using a complexity reduction approach. Genome-based comparisons indicate the bacterium is a novel species within the Sulfuricurvum genus, which contains one cultivated representative, S. kujiense. Divergence between the species appears due in part to extensive genomic rearrangements, gene loss and chromosomal versus plasmid encoding of certain (respiratory) genes by RIFRC-1. Deoxyribonucleic acid for the genome was obtained from terrestrial aquifer sediment, in which RIFRC-1 comprised ∼ 47% of the bacterial community. Genomic evidence suggests RIFRC-1 is a chemolithoautotrophic diazotroph capable of deriving energy for growth by microaerobic or nitrate-/nitric oxide-dependent oxidation of S°, sulfide or sulfite or H₂oxidation. Carbon may be fixed via the reductive tricarboxylic acid cycle. Consistent with these physiological attributes, the local aquifer was microoxic with small concentrations of available nitrate, small but elevated concentrations of reduced sulfur and NH(4)(+) /NH₃-limited. Additionally, various mechanisms for heavy metal and metalloid tolerance and virulence point to a lifestyle well-adapted for metal(loid)-rich environments and a shared evolutionary past with pathogenic Epsilonproteobacteria. Results expand upon recent findings highlighting the potential importance of sulfur and hydrogen metabolism in the terrestrial subsurface.

A RESTful API for accessing microbial community data for MG-RAST

Metagenomic sequencing has produced significant amounts of data in recent years. For example, as of summer 2013, MG-RAST has been used to annotate over 110,000 data sets totaling over 43 Terabases. With metagenomic sequencing finding even wider adoption in the scientific community, the existing web-based analysis tools and infrastructure in MG-RAST provide limited capability for data retrieval and analysis, such as comparative analysis between multiple data sets. Moreover, although the system provides many analysis tools, it is not comprehensive. By opening MG-RAST up via a web services API (application programmers interface) we have greatly expanded access to MG-RAST data, as well as provided a mechanism for the use of third-party analysis tools with MG-RAST data. This RESTful API makes all data and data objects created by the MG-RAST pipeline accessible as JSON objects. As part of the DOE Systems Biology Knowledgebase project (KBase, http://kbase.us) we have implemented a web services API for MG-RAST. This API complements the existing MG-RAST web interface and constitutes the basis of KBases microbial community capabilities. In addition, the API exposes a comprehensive collection of data to programmers. This API, which uses a RESTful (Representational State Transfer) implementation, is compatible with most programming environments and should be easy to use for end users and third parties. It provides comprehensive access to sequence data, quality control results, annotations, and many other data types. Where feasible, we have used standards to expose data and metadata. Code examples are provided in a number of languages both to show the versatility of the API and to provide a starting point for users. We present an API that exposes the data in MG-RAST for consumption by our users, greatly enhancing the utility of the MG-RAST service.