Abel W. Lin

Community cyberinfrastructure for Advanced Microbial Ecology Research and Analysis: the CAMERA resource

The Community Cyberinfrastructure for Advanced Microbial Ecology Research and Analysis (CAMERA, http://camera.calit2.net/) is a database and associated computational infrastructure that provides a single system for depositing, locating, analyzing, visualizing and sharing data about microbial biology through an advanced web-based analysis portal. CAMERA collects and links metadata relevant to environmental metagenome data sets with annotation in a semantically-aware environment allowing users to write expressive semantic queries against the database. To meet the needs of the research community, users are able to query metadata categories such as habitat, sample type, time, location and other environmental physicochemical parameters. CAMERA is compliant with the standards promulgated by the Genomic Standards Consortium (GSC), and sustains a role within the GSC in extending standards for content and format of the metagenomic data and metadata and its submission to the CAMERA repository. To ensure wide, ready access to data and annotation, CAMERA also provides data submission tools to allow researchers to share and forward data to other metagenomics sites and community data archives such as GenBank. It has multiple interfaces for easy submission of large or complex data sets, and supports pre-registration of samples for sequencing. CAMERA integrates a growing list of tools and viewers for querying, analyzing, annotating and comparing metagenome and genome data.

Enabling parallel scientific applications with workflow tools

Electron tomography is a powerful tool for deriving three-dimensional (3D) structural information about biological systems within the spatial scale spanning 1 nm3 and 10 mm3. With this technique, it is possible to derive detailed models of sub-cellular components such as organelles and synaptic complexes and to resolve the 3D distribution of their protein constituents in situ. Due in part to exponentially growing raw data-sizes, there continues to be a need for the increased integration of high-performance computing (HPC) and grid technologies with traditional electron tomography processes to provide faster data processing throughput. This is increasingly relevant because emerging mathematical algorithms that provide better data fidelity are more computationally intensive for larger raw data sizes. Progress has been made towards the transparent use of HPC and grid tools for launching scientific applications without passing on the necessary administrative overhead and complexity (resource administration, authentication, scheduling, data delivery) to the non-computer scientist end-user. There is still a need, however, to simplify the use of these tools for applications developers who are developing novel algorithms for computation. Here we describe the architecture of the Telescience project (http://telescience.ucsd.edu), specifically the use of layered workflow technologies to parallelize and execute scientific codes across a distributed and heterogeneous computational resource pool (including resources from the TeraGrid and OptlPuter projects) without the need for the application developer to understand the intricacies of the grid

CAMERA 2.0: A Data-centric Metagenomics Community Infrastructure Driven by Scientific Workflows

Over the last decade, workflows have been established as a mechanism for scientific developers to create simplified views of complex scientific processes. However, there is a need for a comprehensive system architecture to link scientific developers creating workflows with researchers launching workflows in large scale computing environments. We present the architecture for the CAMERA 2.0 Cyber infrastructure platform that provides a scaffold where workflows can be uploaded into the system, and user interface components for launching and viewing results are automatically generated. In CAMERA 2.0, scientific developers and metagenomics researchers seamlessly collaborate to (i) wrap data-analysis software applications and heterogeneous tools as Resource Oriented Architecture (ROA) components integrating them using scientific workflows; (ii) publish and run scientific workflows via dynamically generated uniform portal interfaces; (iii) map heterogeneous workflow products to provenance and CAMERA semantic database through a transformation component, to save output data resulting from workflow runs based on this mapping; (iv) record and visualize the provenance of all workflow run-related data and processes; and (v) conduct queries across multiple workflow executions and link these workflow executions to each other through data and provenance related to these runs. Furthermore, workflows added to CAMERA also have access to a variety of physical resources for computation and data management. Here, we demonstrate the usability of this framework with some of the developed metagenomics workflows.

Global telescience featuring IPv6 at iGrid2002

Electron tomography is a powerful technique for deriving 3D structural information from biological specimens. As advanced instrumentation, networking, and grid computing are applied to electron tomography and biological sciences in general, much work is needed to integrate and coordinate these advanced technologies in a transparent way to deliver them to the end user. The Telescience Portal (http://gridport.npaci.edu/Telescience) is a web-based solution for end-to-end electron tomography that centralizes applications and seamlessly interfaces with the grid to accelerate the throughput of data results. In this paper we will describe the architecture and design of the Telescience Portal in the context of our experiences leading up to and including the iGrid2002 workshop. We will examine the lessons learned in developing the production Telescience environment, leveraging a successful international collaboration with groups in Japan and Taiwan, building end-to-end native IPv6 networks across continents, and examining IPv6 enabled mechanisms for transferring large data from two unique, remotely accessible high performance scientific instruments. Traditional computer science communities develop next generation technologies. Applications like Telescience drive these next generation technologies into production quality applications for everyday research needs.

Real-time multi-scale brain data acquisition, assembly, and analysis using an end-to-end OptIPuter

At iGrid 2005 we demonstrated the transparent operation of a biology experiment on a test-bed of globally distributed visualization, storage, computational, and network resources. These resources were bundled into a unified platform by utilizing dynamic lambda allocation, high bandwidth protocols for optical networks, a Distributed Virtual Computer (DVC) [N. Taesombut, A. Chien, Distributed Virtual Computer (DVC): Simplifying the development of high performance grid applications, in: Proceedings of the Workshop on Grids and Advanced Networks, GAN 04, Chicago, IL, April 2004 (held in conjunction with the IEEE Cluster Computing and the Grid (CCGrid2004) Conference)], and applications running over the Scalable Adaptive Graphics Environment (SAGE) [L. Renambot, A. Rao, R. Singh, B. Jeong, N. Krishnaprasad, V. Vishwanath, V. Chandrasekhar, N. Schwarz, A. Spale, C. Zhang, G. Goldman, J. Leigh, A. Johnson, SAGE: The Scalable Adaptive Graphics Environment, in: Proceedings of WACE 2004, 23-24 September 2004, Nice, France, 2004]. Using these layered technologies we ran a multi-scale correlated microscopy experiment [M.E. Maryann, T.J. Deerinck, N. Yamada, E. Bushong, H. Ellisman Mark, Correlated 3D light and electron microscopy: Use of high voltage electron microscopy and electron tomography for imaging large biological structures, Journal of Histotechnology 23 (3) (2000) 261-270], where biologists imaged samples with scales ranging from 20X to 5000X in progressively increasing magnification. This allows the scientists to zoom in from entire complex systems such as a rat cerebellum to individual spiny dendrites. The images used spanned multiple modalities of imaging and specimen preparation, thus providing context at every level and allowing the scientists to better understand the biological structures. This demonstration attempts to define an infrastructure based on OptIPuter components which would aid the development and design of collaborative scientific experiments, applications and test-beds and allow the biologists to effectively use the high resolution real estate of tiled displays.

The Telescience Tools: Version 2.0

For over the last decade, researchers at the National Center for Microscopy and Imaging Research (NCMIR, http://ncrrdr.ucsd.edu) have been developing and implementing novel technologies to propel collaborative research. NCMIR has built flexible high throughput, cyber infrastructure environments which connect researchers to advanced imaging instruments datagrids and computational grids. Through the integration of pioneering research in remote instrumentation research (known as Telemicroscopytrade) and grid based distributed computing and data management, this group demonstrated this vision in the context of The Telescience Project (https://telescience.ucsd.edu). Telescience provides an end-to-end, single sign-on solution for biomedical image analysis and structure-function correlation that integrates users with resources and applications with unified security and without prohibitive complexity. Here, we describe the flattening and generalization of the layers of software that make up the Grid-based system architecture of the project. In particular, we focus on the maturation and generalization of the architecture, including the development of an Application to middleware interaction component (ATOMIC) services fabric that streamlines application integration with core grid middleware. This broadening of the overall grid service architecture is the product of over 6 years of user-developer iterative software refinement cycles which have resulted in a suite of Telescience Tools that provide generalized solutions for increasing the interoperability of user interfaces (Web portals and applications) and externally addressable grid resources (instruments and computers)

Case Studies on the Use of Workflow Technologies for Scientific Analysis: The Biomedical Informatics Research Network and the Telescience Project

The advent of “Grids,” or Grid computing, has led to a fundamental shift in the development of applications for managing and performing computational or data-intensive analyses. A current challenge faced by the Grid community entails modeling the work patterns of domain or bench scientists and providing robust solutions utilizing distributed infrastructures. These challenges spawned efforts to develop “workflows” to manage programs and data on behalf of the end user. The technologies come from multiple scientific fields, often with disparate definitions, and have unique advantages and disadvantages, depending on the nature of the scientific process in which they are used. In this chapter, we argue that to maximize the impact of these efforts, there is value in promoting the use of workflows within a tiered, hierarchical structure where each of these emerging workflow pieces are interoperable. We present workflow models of the Telescience™ Project1 and BIRN2 architectures as frameworks that manage multiple tiers of workflows to provide tailored solutions for end-to-end scientific processes.

Scientific Grid Activities and PKI Deployment in the Cybermedia Center, Osaka University

The Cybermedia Center (CMC), Osaka University, is a research institution that offers knowledge and technology resources obtained from advanced researches in the areas of large-scale computation, information and communication, multimedia content and education. Currently, CMC is involved in Japanese national Grid projects such as JGN II (Japan Gigabit Network), NAREGI and BioGrid. Not limited to Japan, CMC also actively takes part in international activities such as PRAGMA. In these projects and international collaborations, CMC has developed a Grid system that allows scientists to perform their analysis by remote-controlling the worlds largest ultra-high voltage electron microscope located in Osaka University. In another undertaking, CMC has assumed a leadership role in BioGrid by sharing its experiences and knowledge on the system development for the area of biology.In this paper, we will give an overview of the BioGrid project and introduce the progress of the Telescience unit, which collaborates with the Telescience Project led by the National Center for Microscopy and Imaging Research (NCMIR). Furthermore, CMC collaborates with seven Computing Centers in Japan, NAREGI and National Institute of Informatics to deploy PKI base authentication infrastructure. The current status of this project and future collaboration with Grid Projects will be delineated in this paper.

The OptIPuter microscopy demonstrator: enabling science through a transatlantic lightpath.

The OptIPuter microscopy demonstrator project has been designed to enable concurrent and remote usage of world-class electron microscopes located in Oxford and San Diego. The project has constructed a network consisting of microscopes and computational and data resources that are all connected by a dedicated network infrastructure using the UK Lightpath and US Starlight systems. Key science drivers include examples from both materials and biological science. The resulting system is now a permanent link between the Oxford and San Diego microscopy centres. This will form the basis of further projects between the sites and expansion of the types of systems that can be remotely controlled, including optical, as well as electron, microscopy. Other improvements will include the updating of the Microsoft cluster software to the high performance computing (HPC) server 2008, which includes the HPC basic profile implementation that will enable the development of interoperable clients.

Case Study on the Use of REST Architectural Principles for Scientific Analysis: CAMERA – Community Cyberinfrastructure for Advanced Microbial Ecology Research and Analysis

The advent of Grid (and by extension Cloud) Computing along with Service Orientated Architecture (SOA) principles have lead to a fundamental shift in the development of end-user application environments. In the scientific domain, this loosely coupled, multi-tiered software architecture has been quickly adopted as raw data sizes have rapidly grown to a point where typical user workstations can no longer perform the necessary computational and data-intensive analyses. A current challenge facing the design and development of SOA involves the management and maintenance of many loosely coupled service components. As with many large applications, “integration” is equally important as “coding”. A resource orientated architecture style serves well in addressing these challenges. Here we present the CAMERA (Community Cyberinfrastructure for Advanced Microbial Ecology Research and Analysis) project as a case study for a SOA in scientific research environments.