Curt Tilmes

Provenance Tracking in an Earth Science Data Processing System

NASA and other organizations involved with climate research have captured huge archives of earth observations. The sensors, spacecraft, and science algorithms for transforming and analyzing the data and the processing frameworks are evolving over time. Science Data Processing Systems (SDPSes) should capture, archive, and distribute provenance information of all externally received data and algorithms, as well as describing all internal processes used for data transformation. This will make the data sets produced by the systems easier to understand, enable independent scientific reproducability, and ultimately, increase the credibility of the scientific research that makes use of those data sets.

Provenance Representation for the National Climate Assessment in the Global Change Information System

The important topic of global climate change builds on a huge collection of scientific research. It is common for agencies releasing climate change information to be served with requests for all supporting materials resulting in a particular conclusion. Capturing and presenting global change provenance, linking to the research papers, data sets, models, analyses, observations, satellites, etc., that support the key research findings in this domain can increase understanding and aid in reproducibility of results and conclusions. The U.S. Global Change Research Program is now coordinating the production of a national climate assessment (NCA) that presents our best understanding of global change. We are now developing a global change information system that will present the content of that report and its provenance, including the scientific support for the findings of the assessment. We are using an approach that will present this information both through a human accessible Web site as well as a machine-readable interface for automated mining of the provenance graph. We plan to use the developing World Wide Web Consortium (W3C) PROV data model and ontology for this system. This paper will describe an overview of the process of developing the NCA and how the provenance trail of the report and each of the technical inputs can be captured and represented using the W3C PROV ontology. This will improve the visibility into the assessment process, increase understanding and possibility of reproducibility, and ultimately increase the credibility and trust of the resulting report.

Development of two Science Investigator-led Processing Systems (SIPS) for NASA's Earth Observation System (EOS)

While building a series of large data processors for remotely sensed data, GSFC LTP generalized the approach and developed a universal system architecture. With a suite of plug-in modules for a variety of functions, the design is a customizable data processing system driven by a rich set of automated production rules. It can take high volumes of diverse inputs, recognize data set types, run many separate processes simultaneously as well as sequentially against the data, and automatically send resulting products to end users. Implemented with commodity hardware and open source software, the highly scalable system has been proven in a number of applications ranging in size from tiny to huge

Service-Oriented Atmospheric Radiances (SOAR): Gridding and Analysis Services for Multisensor Aqua IR Radiance Data for Climate Studies

The Aqua spacecraft, launched on May 4, 2002, carries two well-calibrated independent infrared (IR) grating spectrometers Atmospheric Infrared Sounder (AIRS) and Moderate Resolution Imaging Spectrometer (MODIS), which have been continuously returning upwelling IR spectral radiance measurements for over five years. Based on an Aqua Sr. Project Review, estimates of available flight fuel, power, and orbital projections assess the life span of the Aqua satellite, and these two instruments, to be reliable to 2013. Since launch, these instruments have generated petabytes of data, which are managed and made available by the Goddard Space Flight Center (GSFC) Earth Science Data and Information Services Center and GSFC MODAPS. Agencies such as NOAA, DOD, EPA, and USGS use the AIRS data mostly for weather-related applications, whereas MODIS data are used, in addition to some climate-related studies, for studies of weather, oceans, and land processes, aerosols, natural and man-made disasters, and earth ecology. The Science Investigator-led Processing Systems (SIPS) teams have made many of the desired products derived from these data sets available either as level 2 products and/or level 3 gridded product fields. However, no gridded level 3 data products of radiances, either averaged for a grid element, max, min, or as brightness temperatures (BTs), are provided directly by the SIPS. Thus, one impediment that the general community faces in accessing these MODIS produced petabytes of data is storing such large data sets, interpreting the multiformatted data, and transforming it into helpful data sets for climate-research needs. The Service-Oriented Atmospheric Radiance (SOAR) system has been designed to bridge these gaps and overcome the challenges of bringing this rich data source to the science community, by delivering applications that process these valuable radiance data into standard spatial-temporal grids as well as user-defined criteria on demand. SOAR can serve this community with aggregated, enriched, and thinned gridded data sets provided with access to the data on demand, with query and subsetting capabilities across many dimensions. In addition, SOAR provides online user-specified visualization and analysis requests, all accessible via a Web browser. The utility of SOAR is exposed via Web-service routines, using the Simple Object Access Protocol. The Web-service library and supporting technologies (Axis, PostgreSQL, and Tomcat) reside on a University of Maryland Baltimore Campus client server, which interfaces to and invokes algorithms on the process server, a high-performance computer cluster and storage system. These servers are connected to the sensor data stores at the GSFC via a high-speed fiber-optic network connection [10 Gb/s], providing reliable and fast on-demand access to a vast online library of AIRS and current monthly MODIS source data.

Climate data initiative

Curation is traditionally defined as the process of collecting and organizing information around a common subject matter or a topic of interest and typically occurs in museums, art galleries, and libraries. The task of organizing data around specific topics or themes is a vibrant and growing effort in the biological sciences but to date this effort has not been actively pursued in the Earth sciences. In this paper, we introduce the concept of geocuration and define it as the act of searching, selecting, and synthesizing Earth science data/metadata and information from across disciplines and repositories into a single, cohesive, and useful collection. We present the Climate Data Initiative (CDI) project as a prototypical example. The CDI project is a systematic effort to manually curate and share openly available climate data from various federal agencies. CDI is a broad multi-agency effort of the U.S. government and seeks to leverage the extensive existing federal climate-relevant data to stimulate innovation and private-sector entrepreneurship to support national climate-change preparedness. We describe the geocuration process used in the CDI project, lessons learned, and suggestions to improve similar geocuration efforts in the future.

Distinguishing Provenance Equivalence of Earth Science Data

Abstract Reproducibility of scientific research relies on accurate and precise citation of data and the provenance of that data. Earth science data are often the result of applying complex data transformation and analysis workflows to vast quantities of data. Provenance information of data processing is used for a variety of purposes, including understanding the process and auditing as well as reproducibility. Certain provenance information is essential for producing scientifically equivalent data. Capturing and representing that provenance information and assigning identifiers suitable for precisely distinguishing data granules and datasets is needed for accurate comparisons. This paper discusses scientific equivalence and essential provenance for scientific reproducibility. We use the example of an operational earth science data processing system to illustrate the application of the technique of cascading digital signatures or “hash chains” to precisely identify sets of granules and as provenance equivalence identifiers to distinguish data made in an an equivalent manner.

Implementation of the Land, Atmosphere Near Real-time Capability for EOS (LANCE)

The past decade has seen a rapid increase in availability and usage of near real-time data from satellite sensors. Applications have demonstrated the utility of timely data in a number of areas ranging from numerical weather prediction and forecasting, to monitoring of natural hazards, disaster relief, agriculture and homeland security. As applications mature, the need to transition from prototypes to operational capabilities presents an opportunity to improve current near real-time systems and inform future capabilities. This paper presents NASAs effort to implement a near real-time capability for land and atmosphere data acquired by the Moderate Resolution Imaging Spectroradiometer (MODIS), Atmospheric Infrared Sounder (AIRS), Advanced Microwave Scanning Radiometer - Earth Observing System (AMSR-E), Microwave Limb Sounder (MLS) and Ozone Monitoring Instrument (OMI) instruments on the Terra, Aqua, and Aura satellites.

Concepts for scaling the processing capability of the MODIS data processing System (MODAPS)

The current MODIS processing system (MODAPS), an 80 processor SGI Origin 2000 has proven effective in generating over 500 GB/day of MODIS science products. However its high per processor cost limits the computational resources available for reprocessing these science products. Based on previous missions, the production system should have at least 6 times the capacity needed to keep up with daily production rather than the 2.5 times provided by the current architecture. This paper describes ongoing prototyping studies with clusters of Linux based workstations connected to the central SGI Origin server. A cluster of many inexpensive workstations for swath-based processing connected to a large central server for holding intermediate products and producing global gridded products benefits from both the scalable I/O of the central host, and the cost savings per processor of the commodity workstations.

A Conceptual Enterprise Framework for Managing Scientific Data Stewardship

Scientific data stewardship is an important part of long-term preservation and the use/reuse of digital research data. It is critical for ensuring trustworthiness of data, products, and services, which is important for decision-making. Recent U.S. federal government directives and scientific organization guidelines have levied specific requirements, increasing the need for a more formal approach to ensuring that stewardship activities support compliance verification and reporting. However, many science data centers lack an integrated, systematic, and holistic framework to support such efforts. The current business- and process-oriented stewardship frameworks are too costly and lengthy for most data centers to implement. They often do not explicitly address the federal stewardship requirements and/or the uniqueness of geospatial data. This work proposes a data-centric conceptual enterprise framework for managing stewardship activities, based on the philosophy behind the Plan-Do-Check-Act (PDCA) cycle, a proven industrial concept. This framework, which includes the application of maturity assessment models, allows for quantitative evaluation of how organizations manage their stewardship activities and supports informed decision-making for continual improvement towards full compliance with federal, agency, and user requirements.

Providing provenance to instruments through the US global change information system

The Global Change Information System was created by the US Global Change Research Program to provide specialists and the general public with accessible and usable global change related information. This system uses a relational and semantic web approach to describe the detailed provenance of global change information, such as “in report x, figure y is derived from dataset z”. The system supported the development of the US Third National Climate Assessment, which was released in May 2014 and improves our understanding of climate change in the US. Over the last year, a partnership has been developed with the Committee on Earth Observing Satellites and now their database of missions, instruments and observations is being used to provide traceability to the specific platforms and instruments that support the assessments findings and figures.