Carol Song

HUBzero and Pegasus: integrating scientific workflows into science gateways

In this paper, we described the benefits and the challenges of integrating existing scientific workflow technologies into science gateways. Scientific workflow managers are powerful tools for handling large computational tasks. Domain scientists find it difficult to create new workflows, so many tasks that could benefit from workflow automation are often avoided or performed by hand. Two technologies have come together to bring the benefits of workflow to the masses. The Pegasus Workflow Management System can manage workflows comprised of millions of tasks, all the while recording data about the execution and intermediate results so that the provenance of the final result is clear. The HUBzero platform for scientific collaboration provides a venue for building and delivering tools to researchers and educators. With the press of a button, these tools can launch Pegasus workflows on national computing infrastructures and bring results back for plotting and visualization. As a result, the combination of Pegasus and HUBzero is bringing high‐throughput computing to a much wider audience. Copyright

Implementing an industrial-strength academic cyberinfrastructure at Purdue University

Purdue University operates one of the largest cycle recovery systems in existence in academia based on the Condor workload management system. This system represents a valuable and useful cyberinfrastructure (CI) resource supporting research and education for campus and national users. During the construction and operation of this CI, we encountered many unforeseen challenges and benefits unique to an actively used infrastructure of this size. The most significant problems were integrating Condor with existing campus UPC resources, managing resource and user growth, coping with the distributed ownership of compute resources around campus, and integrating this CI with the Tera- Gridand Open Science Grid. In this paper, we describe some of our experiences and establish some best practices, which we believe will be valuable and useful to other academic institutions seeking to operate a production campus cyberinfrastrucure of a similar scale and utility.

A Policy-Based Accountability Tool for Grid Computing Systems

The dynamic and multi-organizational nature of Grid systems requires effective and efficient accountability systems to scale for accommodating large number of users and resources. The availability of detailed and complete accountability data is crucial for both the Grid administrators and the overall Grid community. In this paper we present a layered architecture for addressing the end-to-end accountability problem. We introduce the concept of accountability agents, entities in charge of collecting accountability data, keeping track of submitted jobs and their users. We present a simple yet effective language to specify the relevant accountability data to be collected and selectively distributed by the accountability agents. Additionally, we design a decentralized and scalable approach to accountability, so to be able to monitor jobs workflow with relatively little intrusion.

iData: a community geospatial data sharing environment to support data-driven science

With the advent of XSEDE, the national cyberinfrastructure has evolved from a set of traditional HPC resources to a broader range of digital services. Science gateways, which serve as portals to scientific applications, have also evolved as researchers are dealing with rapidly expanding scientific datasets and the increasingly complex workflows. More and more gateways are being developed to support integrated services for running data-driven applications on HPC resources such as those on XSEDE. To facilitate this type of workflow, there is a pressing need for web-based data management systems that are easy to use, support data upload, sharing, access and management, and can be integrated with advanced computation and storage resources. More importantly such systems need to be accessible by users from the broad research and education communities. In this paper, we describe the design and implementation of iData, a web-based community data publishing and sharing system. iData supports both generic file-based data collections and several commonly used environmental data collection formats including time series, GIS vector and raster data. Integrated data processing, visualization and filtering capabilities are provided for these data formats. Currently iData can be downloaded and deployed in a HUBzero-based gateway, and we plan to make it available for non-HUBzero platforms in the future. We present two examples in which iData has been successfully used to support research collaboration in driNET and GEOSHARE projects.

Real-time Data Delivery and Remote Visualization through Multi-layer Interfaces

The high resolution NEXRAD level II data provides critical information for researchers and the broader community to understand, monitor, and predict weather in a timely manner. There are several limitations in existing systems for providing easy-to-access 3D visualization of the radar data to the user community. In this paper, we present a scalable and user driven solution for near-real-time remote radar data access, processing, and 3D visualization. The system provides multi-layered interfaces for a broad range of users with different levels of services, allowing them to interactively explore data from multiple radar stations over a time period of interest. Parallel data pre-processing using the Purdue Condor pool and volume caching are implemented to help improve the system response time and scalability. The system also provides reusable radar data services and a set of access points which can be invoked by third party applications. With this hierarchical and user driven design, our system creates a rich and easy to use NEXRAD data service environment for research and education users.

Bring integrated GIS data and modeling capabilities into HUBzero platform

The rapid advancement in the collection of environmental data, geospatial analysis and modeling software calls for new GIS-enabled cyberinfrastructure (CI) capable of support large scale integrated data and modeling activities. Although the HUBzero platform is gaining popularity as a portal development framework for many science and engineering communities, it has limited success in earth and environmental science domains, mainly due to its lack of native support for large geospatial data management and GIS functionalities in the Rappture toolkit. In this paper, we describe our initial work on bringing integrated GIS data and modeling capabilities into the HUBzero platform. We describe the overall architecture of our CI solution which is scalable, extensible and portable to other gateway framework as well. Two prototype applications are presented as a proof-of-concept. Coupling the scalable GIS capabilities with a wealth of tools for user participation and community building on the HUBzero platform, we hope to help new science communities in utilizing this platform.

Domain-specific web services for scientific application developers

Many scientists today routinely conduct simulations and run models using high performance computing (HPC) resources provided by national, regional and campus grid infrastructures, either directly logging into such resources or indirectly via web portals and other application client software, i.e. science gateways. Science gateways have proven to be an effective way of bringing HPC resources to a much larger user base. However, developing a new gateway demands substantial effort, requiring both cyberinfrastructure and domain expertise. This paper describes an approach by core software developers to provide domain specific web services (DomWS) to science gateway and application developers. The core developers deliver a set of web services that understand domain specific semantics and translate domain application requests into computational and other tasks that are dispatched onto the distributed high end resources. This approach allows gateway and application developers to easily create customized application services and focus on the gateway functions and interfaces, such as gadgets and desktop clients, that serve their user community. The DomWS approach is not a generic solution for all simulation services. Our aim is to help application developers apply this framework with ease to their specific models and simulations with a small degree of code modification. DomWS also provides reusable modules and templates for common tasks needed by the targeted user community. This paper describes a set of web services that have been developed for Community Climate System Model (CCSM) version 4 using this approach.

WaterHUB: a resource for students and educators for learning hydrology

The study of surface water hydrology involves understanding the occurrence, distribution and movement of water on the surface of the earth. Because of human impacts in the form of landuse change, the hydrologic processes at one geographic location may be different than other locations under same or different climatic settings. As a result, a tool that educators and students can use to explore hydrology through observed data and computational simulations is needed. The objective of this paper is to present a prototype model sharing and data exploration tool that can be used for education as well as for research. A GIS-enabled model sharing platform for Soil Water Assessment Tool (SWAT), called SWATShare, is developed that enables students to not only run model simulations online, but also publish, share and visualize model results to study the impact of land use change on hydrology at watershed scale. SWATShare is developed as a part of the WaterHUB system that is built by combining Purdues HUBzero technology and TeraGrid/XSEDE computation resources. Experience of developing and using SWATShare in a classroom will be presented and discussed. In addition, the development and implementation plan of another tool, called Hydrology Exploration Tool, will be presented.

Developing an integrated end-to-end TeraGrid climate modeling environment

The Community Earth System Model (CESM) is a widely used community model for studying the climate system on the Earth. The CESM model is both data and computationally intensive, making it difficult for users to set up and run CESM simulations using local resources. In this paper, we describe an integrated climate modeling environment that supports CESM simulations on the TeraGrid, comprehensive model metadata description, and automatic archival of model data and metadata for easy community access. This system builds upon and integrates several existing efforts -- the Purdue CCSM modeling portal, the Earth System Grid, the Earth System Modeling Framework, and the Earth System Curator. We present the design and implementation of our prototype system as well as an end-to-end usage scenario which is broken down into three workflows: model execution, data publishing, and metadata collection/publishing. The system will be used to support research and education on climate systems. We describe our plan and early efforts to engage users and obtain their feedback.

Bringing high performance climate modeling into the classroom

Climate science educators face great challenges on combining theory with hands-on practices in teaching climate modeling. Typical model runs require large computation and storage resources that may not be available on a campus. Additionally, the training and support required to bring novices up to speed would consume significant class time. The same challenges also exist across many other science and engineering disciplines. The TeraGrid science gateway program is leading the way of a new paradigm in addressing such challenges. As part of the TeraGrid science gateway initiative, The Purdue CCSM portal aims at assisting both research and education users to run Community Climate System Model (CCSM) simulations using the TeraGrid high performance computing resources. It provides a one-stop shop for creating, configuring, running CCSM simulations as well as managing jobs and processing output data. The CCSM portal was used in a Purdue graduate class for students to get hands-on experience with running world class climate simulations and use the results to study climate change impact on political policies. The CCSM portal is based on a service-oriented architecture with multiple interfaces to facilitate training. This paper describes the design of the CCSM portal with the goal of supporting classroom users, the challenges of utilizing the portal in a classroom setting, and the solutions implemented. We present two student projects from the fall 2009 class that successfully used the CCSM portal.