Dennis Gannon

TeraGrid Science Gateways and Their Impact on Science

Funded by the National Science Foundation (NSF), TeraGrid is one of the worlds largest distributed cyberinfrastructures for open scientific research. The project began in 2001 as the Distributed Tera-scale Facility, which linked computers, visualization systems, and data at four sites through a dedicated 40-gigabit optical network. Today TeraGrid includes 25 platforms at 11 sites and provides access to more than a petaflop of computing power and petabytes of storage. TeraGrid has three primary focus areas. Its deep goal is to support the most challenging computational science activities those that cannot be achieved without TeraGrid facilities. TeraGrids wide mission is to broaden its user base. The projects open goal is to achieve compatibility with peer grids and information services that allow development of programmatic interfaces to TeraGrid. The Science Gateways program seeks to provide researchers with easy access to TeraGrids high-performance computing resources. A look at four successful gateways illustrates the programs goals, challenges, and opportunities.

Imagining the Future: Thoughts on Computing

New and compelling ideas are transforming the future of computing, bringing about a plethora of changes that have significant implications for our profession and our society and raising some profound technical questions. This Web extra video interview features Dan Reed of Microsoft giving us a sense of how new cloud architectures and cloud capabilities will begin to move computer science education, research, and thinking in whole new directions.

The Client and the Cloud: Democratizing Research Computing

Extending the capabilities of PC, Web, and mobile applications through on-demand cloud services will significantly broaden the research communitys capabilities, accelerating the pace of engineering and scientific discovery in this age of data-driven research. The net effect will be the democratization of research capabilities that are now available only to the most elite scientists.

BioVLAB-Microarray: Microarray Data Analysis in Virtual Environment

Microarray technology is a high-throughput experimental technique that can measure expression levels of hundreds of thousands of genes simultaneously. To interpret massive data from gene-expression microarray experiments, biologists encounter computational and analytical challenges. This is especially challenging for small research labs that lack local computing and bioinformatics expertise. Here, we introduce a virtual analysis system for microarray gene expression data in computing clouds with flexible and configurable GUI workflow engine so that biologists are able to analyze the data in many angles without worrying about computational and bioinformatics issues.

Deadline-sensitive workflow orchestration without explicit resource control

Deadline-sensitive workflows require careful coordination of user constraints with resource availability. Current distributed resource access models provide varying degrees of resource control: from limited or none in grid batch systems to explicit in cloud systems. Additionally applications experience variability due to competing user loads, performance variations, failures, etc. These variations impact the quality of service (QoS) that goes unaccounted for in planning strategies. In this paper we propose Workflow ORchestrator for Distributed Systems (WORDS) architecture based on a least common denominator resource model that abstracts the differences and captures the QoS properties provided by grid and cloud systems. We investigate algorithms for effective orchestration (i.e., resource procurement and task mapping) for deadline-sensitive workflows atop the resource abstraction provided in WORDS. Our evaluation compares orchestration methodologies over TeraGrid and Amazon EC2 systems. Experimental results show that WORDS enables effective orchestration possible at reasonable costs on batch queue grid and cloud systems with or without explicit resource control.

WORKEM: Representing and Emulating Distributed Scientific Workflow Execution State

Scientific workflows have become an integral part of cyber infrastructure as their computational complexity and data sizes have grown. However, the complexity of the distributed infrastructure makes design of new workflows, determining the right management policies, debugging, testing or reproduction of errors challenging. Today, workflow engines manage the dependencies between tasks of workflows and there are tools available to wrap scientific codes. There is a need for a customizable, isolated and manageable testing container for design, evaluation and deployment of distributed workflows. To build such an environment, we need to be able to model and represent, capture and possibly reuse the execution flows within each task of a workflow that accurately captures the execution behavior. In this paper, we present the design and implementation of WORKEM, an extensible framework that can be used to represent and emulate workflow execution state. We also detail the use of the framework in two specific case studies (a) design and testing of an orchestration system (b) generation of a provenance database. Our evaluation shows that the framework has minimal overheads and can be scaled to run hundreds of workflows in short durations of time and with a high amount of parallelism.

Science in the cloud: lessons from three years of research projects on microsoft azure

Microsoft Research is now in its fourth year of awarding Windows Azure cloud resources to the academic community. As of April 2014, over 200 research projects have started. In this paper we review the results of this effort to date. We also characterize the computational paradigms that work well in public cloud environments and those that are usually disappointing. We also discuss many of the barriers to successfully using commercial cloud platforms in research and ways these problems can be overcome.

Towards a Collective Layer in the Big Data Stack

We generalize MapReduce, Iterative MapReduce and data intensive MPI runtime as a layered Map-Collective architecture with Map-All Gather, Map-All Reduce, MapReduce Merge Broadcast and Map-Reduce Scatter patterns as the initial focus. Map-collectives improve the performance and efficiency of the computations while at the same time facilitating ease of use for the users. These collective primitives can be applied to multiple runtimes and we propose building high performance robust implementations that cross cluster and cloud systems. Here we present results for two collectives shared between Hadoop (where we term our extension H-Collectives) on clusters and the Twister4Azure Iterative MapReduce for the Azure Cloud. Our prototype implementations of Map-All Gather and Map-All Reduce primitives achieved up to 33% performance improvement for K-means Clustering and up to 50% improvement for Multi-Dimensional Scaling, while also improving the user friendliness. In some cases, use of Map-collectives virtually eliminated almost all the overheads of the computations.

The Future of Data-Intensive Science

Data-intensive science is now taking a place alongside theoretical science, experimental science, and computational science as a fundamental research paradigm.

Cloud computing architecture and application programming: DISC'09 tutorial, half day, Sept. 22nd 2009

Over the past decade, scientific and engineering research via computing has emerged as the third pillar of the scientific process, complementing theory and experiment. Several studies have highlighted the importance of computational science as a critical enabler of scientific discovery and competitiveness in the physical and biological sciences, medicine and health care, and design and manufacturing. The ability to create rich, detailed models of natural and artificial phenomena and to process large volumes of experimental data, itself created by a new generation of scientific instruments that are themselves powered by computing, makes computing a universal intellectual amplifier, advancing all of science and engineering and powering the knowledge economy. This revolution has been enabled by the availability of inexpensive, powerful processors; low cost, large capacity storage devices; and flexible software tools, each driven by a vibrant consumer and industry marketplace. The explosive growth of research computing systems has created major management, technical and fiscal challenges for both funding agencies and research universities. Typically, faculty members acquire research computing systems, usually small to medium (32–256 nodes) clusters, via research grants and contracts and departmental funds. This distributed acquisition of research computing and inadequate plans for long-term sustainability and technology refresh, mean that universities and funding agencies that support university research, are now struggling to create and maintain compute and data centers to house these systems and to operate and maintain them reliably, in energy-efficient, environmentally friendly ways. Moreover, university budget constraints make efficiency ever more necessary. A growing challenge is satisfying the ever rising demand for research computing and data management the enabler of scientific discovery. Fortuitously, the emergence of cloud computingsoftware and services hosted by networks of commercial data centers and accessible over the Internet offers a solution to this conundrum.