Don Middleton

The Earth System Grid: Supporting the Next Generation of Climate Modeling Research

Understanding the Earths climate system and how it might be changing is a preeminent scientific challenge. Global climate models are used to simulate past, present, and future climates, and experiments are executed continuously on an array of distributed supercomputers. The resulting data archive, spread over several sites, currently contains upwards of 100 TB of simulation data and is growing rapidly. Looking toward mid-decade and beyond, we must anticipate and prepare for distributed climate research data holdings of many petabytes. The Earth System Grid (ESG) is a collaborative interdisciplinary project aimed at addressing the challenge of enabling management, discovery, access, and analysis of these critically important datasets in a distributed and heterogeneous computational environment. The problem is fundamentally a Grid problem. Building upon the Globus toolkit and a variety of other technologies, ESG is developing an environment that addresses authentication, authorization for data access, large-scale data transport and management, services and abstractions for high-performance remote data access, mechanisms for scalable data replication, cataloging with rich semantic and syntactic information, data discovery, distributed monitoring, and Web-based portals for using the system.

Ontology-supported scientific data frameworks: The Virtual Solar-Terrestrial Observatory experience

We have developed a semantic data framework that supports interdisciplinary virtual observatory projects across the fields of solar physics, space physics and solar-terrestrial physics. This work required a formal, machine understandable representation for concepts, relations and attributes of physical quantities in the domains of interest as well as their underlying data representations. To fulfill this need, we developed a set of solar-terrestrial ontologies as formal encodings of the knowledge in the Ontology Web Language-Description Logic (OWL-DL) format. We present our knowledge representation and reasoning needs motivated by the context of Virtual Observatories, from fields spanning upper atmospheric terrestrial physics to solar physics, whose intent is to provide access to observational datasets. The resulting data framework is built upon semantic web methodologies and technologies and provides virtual access to distributed and heterogeneous sets of data as if all resources appear to be organized, stored and retrieved from a local environment. Our conclusion is that the combination of use case-driven, small and modular ontology development, coupled with free and open-source software tools and languages provides sufficient expressiveness and capabilities for an initial production implementation and sets the stage for a more complete semantic-enablement of future frameworks.

High-performance remote access to climate simulation data: a challenge problem for data grid technologies

In numerous scientific disciplines, terabyte and soon petabyte-scale data collections are emerging as critical community resources. A new class of Data Grid infrastructure is required to support management, transport, distributed access to, and analysis of these datasets by potentially thousands of users. Researchers who face this challenge include the Climate Modeling community, which performs long-duration computations accompanied by frequent output of very large files that must be further analyzed. We describe the Earth System Grid prototype, which brings together advanced analysis, replica management, data transfer, request management, and other technologies to support high-performance, interactive analysis of replicated data. We present performance results that demonstrate our ability to manage the location and movement of large datasets from the user’s desktop. We report on experiments conducted over SciNET at SC’2000, where we achieved peak performance of 1.55Gb/s and sustained performance of 512.9Mb/s for data transfers between Texas and California.

An ontology for scientific information in a Grid environment: the earth system Grid

In the emerging world of Grid Computing, shared computational, data, other distributed resources are becoming available to enable scientific advancement through collaborative research and collaboratories. This paper describes the increasing role of ontologies in the context of Grid Computing for obtaining, comparing and analyzing data. We present ontology entities and a declarative model that provide the outline for an ontology of scientific information. Relationships between concepts are also given. The implementation of some concepts described in this ontology is discussed within the context of the Earth System Grid II (ESG)[1].

Data management and analysis for the Earth System Grid

The international climate community is expected to generate hundreds of petabytes of simulation data within the next five to seven years. This data must be accessed and analyzed by thousands of analysts worldwide in order to provide accurate and timely estimates of the likely impact of climate change on physical, biological, and human systems. Climate change is thus not only a scientific challenge of the first order but also a major technological challenge. In order to address this technological challenge, the Earth System Grid Center for Enabling Technologies (ESG-CET) has been established within the U.S. Department of Energys Scientific Discovery through Advanced Computing (SciDAC)-2 program, with support from the offices of Advanced Scientific Computing Research and Biological and Environmental Research. ESG-CETs mission is to provide climate researchers worldwide with access to the data, information, models, analysis tools, and computational capabilities required to make sense of enormous climate simulation datasets. Its specific goals are to (1) make data more useful to climate researchers by developing Grid technology that enhances data usability; (2) meet specific distributed database, data access, and data movement needs of national and international climate projects; (3) provide a universal and secure web-based data access portal for broad multi-model data collections; and (4) provide a wide-range of Grid-enabled climate data analysis tools and diagnostic methods to international climate centers and U.S. government agencies. Building on the successes of the previous Earth System Grid (ESG) project, which has enabled thousands of researchers to access tens of terabytes of data from a small number of ESG sites, ESG-CET is working to integrate a far larger number of distributed data providers, high-bandwidth wide-area networks, and remote computers in a highly collaborative problem-solving environment.

The Earth System Grid Federation: delivering globally accessible petascale data for CMIP5

The fifth Coupled Model Intercomparison Project (CMIP5) will involve the global production and analysis of petabytes of data. The Program for Climate Model Diagnosis and Intercomparison (PCMDI), with responsibility for archival for CMIP5, has established the global “Earth System Grid Federation” (ESGF) of data producers and data archives to support CMIP5. ESGF will provide a set of globally synchronised views of globally distributed data – including some large cache replicants which will be persisted for (at least) decades. Here we describe the archive requirements and key aspects of the resulting architecture. ESGF will stress international networks, as well as the data archives themselves – but significantly less than would have been the case of a centralised archive. Developing and deploying the ESGF has exploited good will and best efforts, but future developments are likely to require more formalised architecture and management.

Building a Global Federation System for Climate Change Research: The Earth System Grid Center for Enabling Technologies (ESG-CET)

The recent release of the Intergovernmental Panel on Climate Change (IPCC) 4th Assessment Report (AR4) has generated significant media attention. Much has been said about the U.S. role in this report, which included significant support from the Department of Energy through the Scientific Discovery through Advanced Computing (SciDAC) and other Department of Energy (DOE) programs for climate model development and the production execution of simulations. The SciDAC-supported Earth System Grid Center for Enabling Technologies (ESG-CET) also played a major role in the IPCC AR4: all of the simulation data that went into the report was made available to climate scientists worldwide exclusively via the ESG-CET. At the same time as the IPCC AR4 database was being developed, the National Center for Atmospheric Research (NCAR), a leading U.S. climate science laboratory and a ESG participant, began publishing model runs from the Community Climate System Model (CCSM), and its predecessor the Parallel Coupled Model (PCM) through ESG. In aggregate, ESG-CET provides seamless access to over 180 terabytes of distributed climate simulation data to over 6,000 registered users worldwide, who have taken delivery of more than 250 terabytes from the archive. Not only does this represent a substantial advance in scientific knowledge, it is also a major step forward in how we conduct the research process on a global scale. Moving forward, the next IPCC assessment report, AR5, will demand multi-site metadata federation for data discovery and cross-domain identity management for single sign- on of users in a more diverse federation enterprise environment. Towards this aim, ESG is leading the effort in the climate community towards standardization of material for the global federation of metadata, security, and data services required to standardize, analyze, and access data worldwide.

Enabling worldwide access to climate simulation data: the earth system grid (ESG)

With support from the U.S. Department of Energys Scientific Discover Through Advanced Computing (SciDAC) program, we have developed and deployed the Earth System Grid (ESG) to make climate simulation data easily accessible to the global climate modelling and analysis community. ESG currently has 2500 registered users and manages 160 TB of data in archives distributed around the nation. From this past year alone, more than 200 scientific journal articles have been published from analyses of data delivered by the ESG.

Enabling Scientific Research using an Interdisciplinary Virtual Observatory: The Virtual Solar-Terrestrial Observatory Example

Our work is aimed at enabling a new style of virtual, distributed scientific research. We have designed, built, and deployed an interdisciplinary virtual observatory—an online service providing access to what appears to be an integrated collection of scientific data. The Virtual Solar-Terrestrial Observatory (VSTO) is a production semantic web data framework providing access to observational data sets from fields spanning upper atmospheric terrestrial physics to solar physics. The observatory allows virtual access to a highly distributed and heterogeneous set of data that appears as if all resources are organized, stored, and retrieved or used in a common way. The end-user community includes scientists, students, and data providers. We will introduce interdisciplinary virtual observatories and their potential impact by describing our experiences with VSTO. We will also highlight some benefits of the embedded semantic web technology and also provide evaluation results after the first year of use.

Data Analysis and Visualization

The Earth’s climate—its average weather and frequency of extreme events—greatly affects the conditions of all living creatures. Human conditions, for example, are strongly influenced by the availability of fresh water, the ambient temperature and humidity, and extreme phenomena such as heat and cold waves, droughts and floods, and tornadoes and hurricanes. Dramatic changes in climatic phenomena (e.g. a Medieval warm period or the prehistoric Ice Ages) also have been recorded, either in human chronicles or in the natural history of the Earth itself.