Lorne Leonard

Essential Terrestrial Variable data workflows for distributed water resources modeling

This paper discusses a prototype infrastructure, HydroTerre, that provides researchers, educators and resource managers with seamless access to geospatial/geotemporal data for supporting physics-based numerical models. The prototype defines the supporting data as Essential Terrestrial Variables (ETVs) and includes data fusion tools necessary to predict and manage surface and groundwater resources that resolve important dynamics of upland stream networks. The evaluation of ecosystem and watershed services, such as the detection and attribution of the impact of climatic change, provides one of many examples of the pressing need for high resolution, spatially explicit resource assessments in upland catchments. However, the current infrastructure for supporting models and data anywhere in the continental USA (CONUS) must overcome important problems of: efficient accessibility to high resolution geospatial datasets from multiple sources, scalability of geospatial data in support of distributed models and data-intensive computation for multi-scale, multi-state simulations. We discuss data workflows for web access to ETV data processing in support of catchment modeling, as part of a larger strategy for consuming this data within a framework that enables hydrological modelers to build and test models with fast data access at a United States Geological Survey (USGS) National Hydrography Dataset Hydrological Unit Code (HUC) level-12 scale. Given the prospect of petabytes of existing high resolution environmental data (NRC, 2012), we limit our investigation to a limited set of ETVs necessary to provide the first level of support for model implementation anywhere in the CONUS, and that resolve important features of upland watersheds (e.g. hill slopes within 1st-2nd-3rd order streams). The paper demonstrates HydroTerre tools for fast ETV data access to web users, and describes the computational resources necessary for using ETVs as the basis for implementing spatially distributed models at scales approaching the native resolution of the data (>=30 m). The Penn State Integrated Hydrologic Model (PIHM) serves as an example although other models are currently being considered.

Automating data-model workflows at a level 12 HUC scale

The prototype discussed in this article retrieves Essential Terrestrial Variable (ETV) web services and uses data-model workflows to transform ETV data for hydrological models in a distributed computing environment. The ETV workflow is a service layer to 100s of terabytes of national datasets bundled for fast data access in support of watershed modeling using the United States Geological Survey (USGS) Hydrological Unit Code (HUC) level-12 scale. The ETV data has been proposed as the Essential Terrestrial Data necessary to construct watershed models anywhere in the continental USA (Leonard and Duffy, 2013). Here, we present the hardware and software system designs to support the ETV, data-model, and model workflows using High Performance Computing (HPC) and service-oriented architecture.This infrastructure design is an important contribution to both how and where the workflows operate. We describe details of how these workflow services operate in a distributed manner for modeling CONUS HUC-12 catchments using the Penn State Integrated Hydrological Model (PIHM) as an example. The prototype is evaluated by generating data-model workflows for every CONUS HUC-12 and creating a repository of workflow provenance for every HUC-12 (~100?km2) for use by researchers as a strategy to begin a new hydrological model study. The concept of provenance for data-model workflows developed here assures reproducibility of model simulations (e.g. reanalysis) from ETV datasets without storing model results which we have shown will require many petabytes of storage. Data-model workflows for reproducibility and provenance of distributed hydrological models.Essential Terrestrial Variable (ETV) datasets used to compute hydrological models.HydroTerre data workflows to create rapid data inputs for HUC-12 catchment scales.Infrastructure to support models and big data at high resolution from multiple federal sources.

Cyber-Innovated Watershed Research at the Shale Hills Critical Zone Observatory

Cyberinfrastructure is enabling ever more integrative and transformative science. Technological advances in cyberinfrastructure have allowed deeper understanding of watershed hydrology by improved integration of data, information, and models. The synthesis of all sources of hydrologic variables (historical, real time, future scenarios, observed, and modeled) requires advanced data acquisition, data storage, data management, data integration, data mining, and data visualization. In this context, cyber-innovated hydrologic research was implemented to carry out watershed-based historical climate simulations at the Shale Hills Critical Zone Observatory. The simulations were based on the assimilation of data from a hydrologic monitoring network into a multiphysics hydrologic model (the Penn State Integrated Hydrology Model). We documented workflows for the model application and applied the model to short-time hyporheic exchange flow study and long-term climate scenario analysis. The effort reported herein demonstrates that advances in cyberscience allows innovative research that improves our ability to access and share data; to allow collective development of science hypotheses; and to support building models via team participation. We simplified communications between model developers and community scientists, software professionals, students, and decision makers, which in the long term will improve the utilization of hydrologic models for science and societal applications.

Visualization workflows for level-12 HUC scales

Visualization workflows are important services for expert users to analyze watersheds when using our HydroTerre end-to-end workflows. Analysis is an interactive and iterative process and we demonstrate that the expert user can focus on model results, not data preparation, by using a web application to rapidly create, tune, and calibrate hydrological models anywhere in the continental USA (CONUS). The HydroTerre system captures user interaction for provenance and reproducibility to share modeling strategies with modelers. Our end-to-end workflow consists of four workflows. The first is data workflows using Essential Terrestrial Variables (ETV) data sets that we demonstrated to construct watershed models anywhere in the CONUS (Leonard and Duffy, 2013). The second is data-model workflows that transform the data workflow results to model inputs. The model inputs are consumed in the third workflow, model workflows (Leonard and Duffy, 2014a) that handle distribution of data and model within High Performance Computing (HPC) environments. This article focuses on our fourth workflow, visualization workflows, which consume the first three workflows to form an end-to-end system to create and share hydrological model results efficiently for analysis and peer review. We show how visualization workflows are incorporated into the HydroTerre infrastructure design and demonstrate the efficiency and robustness for an expert modeler to produce, analyze, and share new hydrological models using CONUS national datasets. Visualization, model, and data services for and sharing of hydrological models.Visualization workflows for hydrological model analysis at HUC12 catchment scales.End-to-end workflows for reproducibility and provenance of hydrological models.Cyber-Infrastructure to support hydrological models and big data at high resolution.

Tuning Heterogeneous Computing Platforms for Large-Scale Hydrology Data Management

HydroTerre is a research prototype platform developed at Penn State for the hydrology community. It provides access to aggregated scientific data sets that are useful for hydrological modeling and research. HydroTerres frontend is a web service, and a user query can request creation of a data bundle whose size can vary from a few megabytes to 100s of gigabytes. In this article, we present software tuning and optimization strategies for various hardware configurations of the HydroTerre platform. Our goal is to minimize access time to a wide range of data bundle creation queries from users. We use automated schemes to estimate the computational work required for various queries, and identify the best-performing hardware/software configuration. We hope this study is instructive for researchers developing similar data management cyberinfrastructure in other science and engineering fields.

Graph-based visual analysis for large-scale hydrological modeling

This article reports on the development and application of a visual analytics approach to big data cleaning and integration focused on very large graphs, constructed in support of national-scale hydrological modeling. We explain why large graphs are required for hydrology modeling and describe how we create two graphs using continental United States heterogeneous national data products. The first smaller graph is constructed by assigning level-12 hydrological unit code watersheds as nodes. Creating and cleaning graphs at this scale highlight the issues that cannot be addressed without high-resolution datasets and expert intervention. Expert intervention, aided with visual analytical tools, is necessary to address edge directions at the second graph scale: subdividing continental United States streams as edges (851,265,305) and nodes (683,298,991) for large-scale hydrological modeling. We demonstrate how large graph workflows are created and are used for automated analysis to prepare the user interface for visual analytics. We explain the design of the visual interface using a watershed case study and then discuss how the visual interface is used to engage the expert user to resolve data and graph issues.

Development of a participatory Green Infrastructure design, visualization and evaluation system in a cloud supported jupyter notebook computing environment

Abstract Land use planners, landscape architects, and water resource managers are using Green Infrastructure (GI) designs in urban environments to promote ecosystem services including mitigation of storm water flooding and water quality degradation. An expanded set of urban sustainability goals also includes increasing carbon sequestration, songbird habitat, reducing urban heat island effects, and improvement of landscape aesthetics. GI is conceptualized to improve water and ecosystem quality by reducing storm water runoff at the source, but when properly designed, may also benefit these expanded goals. With the increasing use of GI in urban contexts, there is an emerging need to facilitate participatory design and scenario evaluation to enable better communication between GI designers and groups impacted by these designs. Major barriers to this type of public participation is the complexity of both parameterizing, operating, visualizing and interpreting results of complex ecohydrological models at various watershed scales that are sufficient to address diverse ecosystem service goals. This paper demonstrates a set of workflows to facilitate rapid and repeatable creation of GI landscape designs which are incorporated into complex models using web applications and services. For this project, we use the RHESSys (Regional Hydro-Ecologic Simulation System) ecohydrologic model to evaluate participatory GI landscape designs generated by stakeholders and decision makers, but note that the workflow could be adapted to a set of other watershed models.

EAKOS: VAST 2009

In this article, I describe the tools and techniques used to generate competing hypotheses for the VAST 2009 Flitter mini challenge. I will describe how I approached solving the social networks and the importance of the geospatial relationships to determine that ldquoSocial Structure Form Ardquo was the best matching social network.