Dean N. Williams

The Earth System Grid: Supporting the Next Generation of Climate Modeling Research

Understanding the Earths climate system and how it might be changing is a preeminent scientific challenge. Global climate models are used to simulate past, present, and future climates, and experiments are executed continuously on an array of distributed supercomputers. The resulting data archive, spread over several sites, currently contains upwards of 100 TB of simulation data and is growing rapidly. Looking toward mid-decade and beyond, we must anticipate and prepare for distributed climate research data holdings of many petabytes. The Earth System Grid (ESG) is a collaborative interdisciplinary project aimed at addressing the challenge of enabling management, discovery, access, and analysis of these critically important datasets in a distributed and heterogeneous computational environment. The problem is fundamentally a Grid problem. Building upon the Globus toolkit and a variety of other technologies, ESG is developing an environment that addresses authentication, authorization for data access, large-scale data transport and management, services and abstractions for high-performance remote data access, mechanisms for scalable data replication, cataloging with rich semantic and syntactic information, data discovery, distributed monitoring, and Web-based portals for using the system.

Ensemble-Vis: A Framework for the Statistical Visualization of Ensemble Data

Scientists increasingly use ensemble data sets to explore relationships present in dynamic systems. Ensemble data sets combine spatio-temporal simulation results generated using multiple numerical models, sampled input conditions and perturbed parameters. While ensemble data sets are a powerful tool for mitigating uncertainty, they pose significant visualization and analysis challenges due to their complexity. In this article, we present Ensemble-Vis, a framework consisting of a collection of overview and statistical displays linked through a high level of interactivity. Ensemble-Vis allows scientists to gain key scientific insight into the distribution of simulation results as well as the uncertainty associated with the scientific data. In contrast to methods that present large amounts of diverse information in a single display, we argue that combining multiple linked displays yields a clearer presentation of the data and facilitates a greater level of visual data analysis. We demonstrate our framework using driving problems from climate modeling and meteorology and discuss generalizations to other fields.

The Earth System Grid: Enabling Access to Multimodel Climate Simulation Data

By leveraging current technologies to manage distributed climate data in a unified virtual environment, the Earth System Grid (ESG) project is promoting data sharing between international research centers and diverse users. In transforming these data into a collaborative community resource, ESG is changing the way global climate research is conducted. Since ESGs production beginnings in 2004, its most notable accomplishment was to efficiently store and distribute climate simulation data of some 20 global coupled ocean-atmosphere models to the scores of scientific contributors to the Fourth Assessment Report (AR4) of the Intergovernmental Panel on Climate Change (IPCC); the IPCC collective scientific achievement was recognized by the award of a 2007 Nobel Peace Prize. Other international climate stakeholders such as the North American Regional Climate Change Assessment Program (NARCCAP) and the developers of the Community Climate System Model (CCSM) and of the Climate Science Computational End Station (CC...

Visualizing wind velocities by advecting cloud textures

In order to visualize both clouds and wind in climate simulations, clouds were rendered using a 3D texture which was advected by the wind flow. The simulation is described. Rendering, the advection of texture coordinates, and haze effects are discussed. Results are presented.<<ETX>>

The Earth System Grid Federation: An open infrastructure for access to distributed geospatial data

Abstract The Earth System Grid Federation (ESGF) is a multi-agency, international collaboration that aims at developing the software infrastructure needed to facilitate and empower the study of climate change on a global scale. The ESGF’s architecture employs a system of geographically distributed peer nodes, which are independently administered yet united by the adoption of common federation protocols and application programming interfaces (APIs). The cornerstones of its interoperability are the peer-to-peer messaging that is continuously exchanged among all nodes in the federation; a shared architecture and API for search and discovery; and a security infrastructure based on industry standards (OpenID, SSL, GSI and SAML). The ESGF software stack integrates custom components (for data publishing, searching, user interface, security and messaging), developed collaboratively by the team, with popular application engines (Tomcat, Solr) available from the open source community. The full ESGF infrastructure has now been adopted by multiple Earth science projects and allows access to petabytes of geophysical data, including the entire Fifth Coupled Model Intercomparison Project (CMIP5) output used by the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) and a suite of satellite observations (obs4MIPs) and reanalysis data sets (ANA4MIPs). This paper presents ESGF as a successful example of integration of disparate open source technologies into a cohesive, wide functional system, and describes our experience in building and operating a distributed and federated infrastructure to serve the needs of the global climate science community.

A Provenance-Based Infrastructure to Support the Life Cycle of Executable Papers

As publishers establish a greater online presence as well as infrastructure to support the distribution of more varied information, the idea of an executable paper that enables greater interaction has developed. An executable paper provides more information for computational experiments and results than the text, tables, and figures of standard papers. Executable papers can bundle computational content that allow readers and reviewers to interact, validate, and explore experiments. By including such content, authors facilitate future discoveries by lowering the barrier to reproducing and extending results. We present an infrastructure for creating, disseminating, and maintaining executable papers. Our approach is rooted in provenance, the documentation of exactly how data, experiments, and results were generated. We seek to improve the experience for everyone involved in the life cycle of an executable paper. The automated capture of provenance information allows authors to easily integrate and update results into papers as they write, and also helps reviewers better evaluate approaches by enabling them to explore experimental results by varying parameters or data. With a provenance-based system, readers are able to examine exactly how a result was developed to better understand and extend published findings.

High-performance remote access to climate simulation data: a challenge problem for data grid technologies

In numerous scientific disciplines, terabyte and soon petabyte-scale data collections are emerging as critical community resources. A new class of Data Grid infrastructure is required to support management, transport, distributed access to, and analysis of these datasets by potentially thousands of users. Researchers who face this challenge include the Climate Modeling community, which performs long-duration computations accompanied by frequent output of very large files that must be further analyzed. We describe the Earth System Grid prototype, which brings together advanced analysis, replica management, data transfer, request management, and other technologies to support high-performance, interactive analysis of replicated data. We present performance results that demonstrate our ability to manage the location and movement of large datasets from the user’s desktop. We report on experiments conducted over SciNET at SC’2000, where we achieved peak performance of 1.55Gb/s and sustained performance of 512.9Mb/s for data transfers between Texas and California.

Visualization for climate modeling

Techniques for visualizing several climate variables, particularly clouds and wind velocities, are discussed. The techniques are volume rendering, textured contour surfaces, and vector field rendering. The application of these techniques is discussed in the context of a high-definition television (HDTV) animation produced to show global cloud motion during ten simulated days in January. The computational grid is defined by nearly a million grid points-320 evenly spaced longitudes, 160 unevenly spaced latitudes, and 19 unevenly spaced surfaces of constant geopotential. The climate variables at every grid point were output at every hour of simulated time, resulting in 380 MB of data per simulated day.<<ETX>>

Ophidia: toward big data analytics for eScience

This work introduces Ophidia, a big data analytics research effort aiming at supporting the access, analysis and mining of scientific (n-dimensional array based) data. The Ophidia platform extends, in terms of both primitives and data types, current relational database system implementations (in particular MySQL) to enable efficient data analysis tasks on scientific array-based data. To enable big data analytics it exploits well-known scientific numerical libraries, a distributed and hierarchical storage model and a parallel software framework based on the Message Passing Interface to run from single tasks to more complex dataflows. The current version of the Ophidia platform is being tested on NetCDF data produced by CMCC climate scientists in the context of the international Coupled Model Intercomparison Project Phase 5 (CMIP5).

Visualization of uncertainty and ensemble data: Exploration of climate modeling and weather forecast data with integrated ViSUS-CDAT systems

Climate scientists and meteorologists are working towards a better understanding of atmospheric conditions and global climate change. To explore the relationships present in numerical predictions of the atmosphere, ensemble datasets are produced that combine time- and spatially-varying simulations generated using multiple numeric models, sampled input conditions, and perturbed parameters. These data sets mitigate as well as describe the uncertainty present in the data by providing insight into the effects of parameter perturbation, sensitivity to initial conditions, and inconsistencies in model outcomes. As such, massive amounts of data are produced, creating challenges both in data analysis and in visualization. This work presents an approach to understanding ensembles by using a collection of statistical descriptors to summarize the data, and displaying these descriptors using variety of visualization techniques which are familiar to domain experts. The resulting techniques are integrated into the ViSUS/Climate Data and Analysis Tools (CDAT) system designed to provide a directly accessible, complex visualization framework to atmospheric researchers.