Paul M. Ramirez

A Reusable Process Control System Framework for the Orbiting Carbon Observatory and NPP Sounder PEATE Missions

We describe a reusable architecture and implementation framework for managing science processing pipelines for mission ground data systems. Our system, dubbed ``PCS, for Process Control System, improves upon an existing software component, the OODT Catalog and Archive (CAS), which has already supported the QuikSCAT, SeaWinds and AMT earth science missions. This paper focuses on PCS within the context of two current earth science missions: the Orbiting Carbon Observatory (OCO), and NPP Sounder PEATE projects.

SciSpark: Applying in-memory distributed computing to weather event detection and tracking

In this paper we present SciSpark, a Big Data framework that extends Apache™ Spark for scaling scientific computations. The paper details the initial architecture and design of SciSpark. We demonstrate how SciSpark achieves parallel ingesting and partitioning of earth science satellite and model datasets. We also illustrate the usability and extensibility of SciSpark by implementing aspects of the Grab em Tag em Graph em (GTG) algorithm using SciSpark and its Map Reduce capabilities. GTG is a topical automated method for identifying and tracking Mesoscale Convective Complexes in satellite infrared datasets.

Cloud computing and virtualization within the regional climate model and evaluation system

The Regional Climate Model Evaluation System (RCMES) facilitates the rapid, flexible inclusion of NASA observations into climate model evaluations. RCMES provides two fundamental components. A database (RCMED) is a scalable point-oriented cloud database used to elastically store remote sensing observations and to make them available using a space time query interface. The analysis toolkit (RCMET) is a Python-based toolkit that can be delivered as a cloud virtual machine, or as an installer package deployed using Python Buildout to users in order to allow for temporal and spatial regridding, metrics calculation (RMSE, bias, PDFs, etc.) and end-user visualization. RCMET is available to users in an “offline”, lone scientist mode based on a virtual machine dynamically constructed with model outputs and observations to evaluate; or on an institution’s computational cluster seated close to the observations and model outputs. We have leveraged RCMES within the content of the Coordinated Regional Downscaling Experiment (CORDEX) project, working with the University of Cape Town and other institutions to compare the model output to NASA remote sensing data; in addition we are also working with the North American Regional Climate Change Assessment Program (NARCCAP). In this paper we explain the contribution of cloud computing to RCMES’s specifically describing studies of various cloud databases we evaluated for RCMED, and virtualization toolkits for RCMET, and their potential strengths in delivering user-created dynamic regional climate model evaluation virtual machines for our users.

Exploring a graph theory based algorithm for automated identification and characterization of large mesoscale convective systems in satellite datasets

Mesoscale convective systems are high impact convectively driven weather systems that contribute large amounts to the precipitation daily and monthly totals at various locations globally. As such, an understanding of the lifecycle, characteristics, frequency and seasonality of these convective features is important for several sectors and studies in climate studies, agricultural and hydrological studies, and disaster management. This study explores the applicability of graph theory to creating a fully automated algorithm for identifying mesoscale convective systems and determining their precipitation characteristics from satellite datasets. Our results show that applying graph theory to this problem allows for the identification of features from infrared satellite data and the seamlessly identification in a precipitation rate satellite-based dataset, while innately handling the inherent complexity and non-linearity of mesoscale convective systems.

Developing an open source strategy for NASA earth science data systems

We have found open source to be an effective platform for software reuse. Within the NASA Earth science data systems community, there are a number of distinct applications, ranging from interactions amongst science investigator led processing systems (SIPS), which focus on active data processing, algorithm experimentation and evaluation, and the reuse of instrument processing approaches; to NASAs Distributed Active Archive Centers (DAACs) that are responsible for outward facing data dissemination to the public, and where long term preservation of data and reuse are distinctly important; all the way to downstream proposal led systems, where investigators are funded by NASA to reuse data and software to produce fused data products, and to aggregate and reuse NASA data systems in a systems-of-systems manner. Recognizing the need for a coordinated effort to inform the reuse of components within the NASA ecosystem, we are developing a strategic approach for the development and reuse of open source software. The NASA open source strategy builds on a set of dimensions involving legal, architectural, community, and redistribution areas that are of prime importance to the agency as a whole.

24 Hour near real time processing and computation for the JPL Airborne Snow Observatory

JPLs Airborne Snow Observatory is an integrated imaging spectrometer and scanning LIDAR for measuring mountain snow albedo, snow depth/snow water equivalent, and ice height (once exposed). This paper describes the first year of the projects Snow On campaign where over a course of 3 months, ASO flew the Tuolumne River Basin, Sierra Nevada, California above the OShaughnessy Dam of the Hetch Hetchy reservoir; focusing initial on the Tuolumne, and then moved to weekly flights over the Uncompahgre Basin, Colorado. To meet the needs of its customers including Water Resource managers who are keenly interested in Snow melt, the ASO team had to develop and end to end 24 hour latency capability for processing spectrometer and LIDAR data from Level 0 to Level 4 products. This paper describes the Big data processing architecture and data system for ASO.

PDS4: A model-driven planetary science data architecture for long-term preservation

The goal of the Planetary Data System (PDS) is the digital preservation of scientific data for long-term use by the scientific research community. After two decades of successful operation, the PDS found itself in a new era of big data, international cooperation, distributed nodes, and multiple ways of analysing and interpreting data. A project was formed to develop a disciplined architectural approach that would drive the design and implementation of a scalable data system that could evolve to meet the demands of this new era. PDS4, the next generation system, uses an explicit model-driven architectural approach coupled with modern information technologies and standards to meet these challenges in order to ensure the planetary data assets can be mined for scientific knowledge for years to come.

SciSpark: Highly interactive in-memory science data analytics

We present further work on SciSpark, a Big Data framework that extends Apache Sparks inmemory parallel computing to scale scientific computations. SciSparks current architecture and design includes: time and space partitioning of highresolution geo-grids from NetCDF3/4; a sciDataset class providing N-dimensional array operations in Scala/Java and CF-style variable attributes (an update of our prior sciTensor class); parallel computation of time-series statistical metrics; and an interactive front-end using science (code & visualization) Notebooks. We demonstrate how SciSpark achieves parallel ingest and time/space partitioning of Earth science satellite and model datasets. We illustrate the usability, extensibility, and early performance of SciSpark using several Earth science Use cases, here presenting benchmarks for sciDataset Readers and parallel time-series analytics. A three-hour SciSpark tutorial was taught at an ESIP Federation meeting using a dozen “live” Notebooks.

Developing an open source, reusable platform for distributed collaborative information management in the Early Detection Research Network

For the past decade, the NASA Jet Propulsion Laboratory, in collaboration with Dartmouth University has served as the center for informatics for the Early Detection Research Network (EDRN). The EDRN is a multi-institution research effort funded by the U.S. National Cancer Institute (NCI) and tasked with identifying and validating biomarkers for the early detection of cancer. As the distributed network has grown, increasingly formal processes have been developed for the acquisition, curation, storage, and dissemination of heterogeneous research information assets, and an informatics infrastructure has emerged. In this paper we discuss the evolution of EDRN informatics, its success as a mechanism for distributed information integration, and the potential sustainability and reuse benefits of emerging efforts to make the platform components themselves open source. We describe our experience transitioning a large closed-source software system to a community-driven, open source project at the Apache Software Foundation, and point to lessons learned that will guide our present efforts to promote the reuse of the EDRN informatics infrastructure by a broader community.

Radio Array of Portable Interferometric Detectors (RAPID): Design and applications

The Radio Array of Portable Interferometric Detectors (RAPID) is a spatially diverse sparse radio array. It has been designed to be deployed and reconfigured easily for scientific applications. These applications include both geospace and astronomy experiments where a relatively small and sparse aperture is sufficient in size. Examples include the study of ionospheric turbulence using active and passive radar imaging, astronomical observations of galactic synchrotron emission, and localization of bright radio emissions such as those from the Sun and Jupiter. The high degree of mobility afforded by the system enables interferometric configurations that are tailored to specific experiments and can be changed in the field. RAPID can also be deployed to locations that are optimal for specific scientific objectives or that complement other existing facilities by adding baselines or serving as a separate imaging receiver array.