Jeffery S. Horsburgh

An integrated system for publishing environmental observations data

Over the next decade, it is likely that science and engineering research will produce more scientific data than has been created over the whole of human history. The successful use of these data to achieve new scientific breakthroughs will depend on the ability to access, integrate, and analyze these large datasets. Robust data organization and publication methods are needed within the research community to enable data discovery and scientific analysis by researchers other than those that collected the data. We present a new method for publishing research datasets consisting of point observations that employs a standard observations data model populated using controlled vocabularies for environmental and water resources data along with web services for transmitting data to consumers. We describe how these components have reduced the syntactic and semantic heterogeneity in the data assembled within a national network of environmental observatory test beds and how this data publication system has been used to create a federated network of consistent research data out of a set of geographically decentralized and autonomous test bed databases.

HydroDesktop: Web services-based software for hydrologic data discovery, download, visualization, and analysis

Discovering and accessing hydrologic and climate data for use in research or water management can be a difficult task that consumes valuable time and personnel resources. Until recently, this task required discovering and navigating many different data repositories, each having its own website, query interface, data formats, and descriptive language. New advances in cyberinfrastructure and in semantic mediation technologies have provided the means for creating better tools supporting data discovery and access. In this paper we describe a freely available and open source software tool, called HydroDesktop, that can be used for discovering, downloading, managing, visualizing, and analyzing hydrologic data. HydroDesktop was created as a means for searching across and accessing hydrologic data services that have been published using the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) Hydrologic Information System (HIS). We describe the design and architecture of HydroDesktop, its novel contributions in web services-based hydrologic data search and discovery, and its unique extensibility interface that enables developers to create custom data analysis and visualization plug-ins. The functionality of HydroDesktop and some of its existing plug-ins are introduced in the context of a case study for discovering, downloading, and visualizing data within the Bear River Watershed in Idaho, USA.

A sensor network for high frequency estimation of water quality constituent fluxes using surrogates

Characterizing spatial and temporal variability in the fluxes and stores of water and water borne constituents is important in understanding the mechanisms and flow paths that carry constituents to a stream and through a watershed. High frequency data collected at multiple sites can be used to more effectively quantify spatial and temporal variability in water quality constituent fluxes than through the use of low frequency water quality grab sampling. However, for many constituents (e.g., sediment and phosphorus) in-situ sensor technology does not currently exist for making high frequency measurements of constituent concentrations. In this paper we describe how water quality measures such as turbidity or specific conductance, which can be measured in-situ with high frequency, can be used as surrogates for other water quality constituents that cannot economically be measured with high frequency to provide continuous time series of water quality constituent concentrations and fluxes. We describe the observing infrastructure required to make high frequency estimates of water quality constituent fluxes based on surrogate data at multiple sites within a sensor network supporting an environmental observatory. This includes the supporting sensor, communication, data management, and data storage and processing infrastructure. We then provide a case study implementation in the Little Bear River watershed of northern Utah, USA, where a wireless sensor network has been developed for estimating total phosphorus and total suspended solids fluxes using turbidity as a surrogate.

HydroShare: Sharing Diverse Environmental Data Types and Models as Social Objects with Application to the Hydrology Domain

The types of data and models used within the hydrologic science community are diverse. New repositories have succeeded in making data and models more accessible, but are, in most cases, limited to particular types or classes of data or models and also lack the type of collaborative and iterative functionality needed to enable shared data collection and modeling workflows. File sharing systems currently used within many scientific communities for private sharing of preliminary and intermediate data and modeling products do not support collaborative data capture, description, visualization, and annotation. In this article, we cast hydrologic datasets and models as “social objects” that can be published, collaborated around, annotated, discovered, and accessed. This article describes the generic data model and content packaging scheme for diverse hydrologic datasets and models used by a new hydrologic collaborative environment called HydroShare to enable storage, management, sharing, publication, and annotation of the diverse types of data and models used by hydrologic scientists. The flexibility of HydroShares data model and packaging scheme is demonstrated using multiple hydrologic data and model use cases that highlight its features.

Data visualization and analysis within a Hydrologic Information System: Integrating with the R statistical computing environment

This paper presents a prototype software system for visualization and analysis of hydrologic data that provides interoperability between the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) Hydrologic Information System (HIS) and the R statistical computing environment. By linking these two systems within a single desktop software application, an integrated hydrologic data management and analysis environment has been created that simplifies the process used by scientists and engineers to find, access, organize, and analyze the hydrologic data needed in modeling and managing hydrologic and environmental systems. The implementation of this work is a software plug-in for the CUAHSI HIS HydroDesktop software system called HydroR. We describe the design, graphical user interface, and implementation of the HydroR plug-in. An example application of HydroR is presented in which total suspended solids concentrations are modeled for the Little Bear River using a regression developed from turbidity and total suspended solids observations downloaded from the CUAHSI HIS using HydroDesktop. Finally, we conclude with a summary of our experience in developing interoperability between HIS and R and suggest future developments that can extend the capabilities we have developed. We developed a software system that integrates the CUAHSI HIS and R.HydroR combines data discovery and management with analysis and visualization in a single software environment.HydroR reduces effort required to access data and transfer it into an analysis environment.HydroR promotes repeatable analyses and may reduce potential errors.

iSAW: Integrating Structure, Actors, and Water to Study Socio-Hydro-Ecological Systems

Urbanization, climate, and ecosystem change represent major challenges for managing water resources. Although water systems are complex, a need exists for a generalized representation of these systems to identify important components and linkages to guide scientific inquiry and aid water management. We developed an integrated Structure-Actor-Water framework (iSAW) to facilitate the understanding of and transitions to sustainable water systems. Our goal was to produce an interdisciplinary framework for water resources research that could address management challenges across scales (e.g., plot to region) and domains (e.g., water supply and quality, transitioning, and urban landscapes). The framework was designed to be generalizable across all human–environment systems, yet with sufficient detail and flexibility to be customized to specific cases. iSAW includes three major components: structure (natural, built, and social), actors (individual and organizational), and water (quality and quantity). Key linkages among these components include: (1) ecological/hydrologic processes, (2) ecosystem/geomorphic feedbacks, (3) planning, design, and policy, (4) perceptions, information, and experience, (5) resource access and risk, and (6) operational water use and management. We illustrate the flexibility and utility of the iSAW framework by applying it to two research and management problems: understanding urban water supply and demand in a changing climate and expanding use of green storm water infrastructure in a semi-arid environment. The applications demonstrate that a generalized conceptual model can identify important components and linkages in complex and diverse water systems and facilitate communication about those systems among researchers from diverse disciplines.

Open source software for visualization and quality control of continuous hydrologic and water quality sensor data

It is common for in situ hydrologic and water quality data to be collected at high frequencies and for extended durations. These data streams, which may also be collected across many monitoring sites require infrastructure for data storage and management. The Observations Data Model (ODM), which is part of the Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) Hydrologic Information System (HIS), was developed as a standard data model in which to organize, store, and describe point observations data. In this paper we describe ODM Tools Python, an open source software application that allows users to query and export, visualize, and perform quality control post processing on time series of environmental observations data stored in an ODM database using automated Python scripting that records the corrections and adjustments made to data series in the quality control process and ensures data editing steps are traceable and reproducible. We developed a workflow for scripting of time series data quality control.ODM Tools automatically scripts manual quality control data edits in Python.ODM Tools preserves provenance of quality control edits.ODM Tools is open source and cross platform compatible.ODM Tools demonstrates options for desktop application development and deployment using Python.

Managing a community shared vocabulary for hydrologic observations

The ability to discover and integrate data from multiple sources, projects, and research efforts is critical as scientists continue to investigate complex hydrologic processes at expanding spatial and temporal scales. Until recently, syntactic and semantic heterogeneity in data from different sources made data discovery and integration difficult. The Consortium of Universities for the Advancement of Hydrologic Science, Inc. (CUAHSI) Hydrologic Information System (HIS) was developed to improve access to hydrologic data. A major semantic challenge related to data sharing and publication arose in development of the HIS. No accepted vocabulary existed within the hydrology research community for describing hydrologic observations, making it difficult to discover and synthesize data from multiple research groups even if access to the data was not a barrier. Additionally, the hydrology research community relies heavily on data collected or assembled by government agencies such as USGS and USEPA, each of which has its own semantics for describing observations. This semantic heterogeneity across data sources was a challenge in developing tools that support data discovery and access across multiple hydrologic data sources by time, geographic region, measured variable, data collection method, etc. This paper describes a community shared vocabulary and its supporting management tools that can be used by data publishers to populate metadata describing hydrologic observations to ensure that data from multiple sources published within the CUAHSI HIS are semantically consistent. We also describe how the CUAHSI HIS mediates across terms in the community shared vocabulary and terms used by government agencies to support discovery and integration of datasets published by both academic researchers and government agencies. The CUAHSI Hydrologic Information System provides a community vocabulary for describing hydrologic observations.The community vocabulary promotes semantic consistency in published observations.The community vocabulary simplifies use of variable names and other attributes.The community vocabulary supports data publication, discovery, and interpretation.Software for community maintenance of the vocabulary is described.

Evaluating the simulation times and mass balance errors of component-based models

In making the decision whether to use component-based modeling, its benefits must be balanced against computational costs. Studies evaluating these costs using the Open Modeling Interface (OpenMI) have largely used models with simplified formulations, small spatial and temporal domains, or a limited number of components. We evaluate these costs by applying OpenMI to a relatively complex Stormwater Management Model (SWMM) for the City of Logan, Utah, USA. Configurations of coupled OpenMI components resulting from decomposing the stormwater model by process (i.e., runoff coupled to routing) and then by space (i.e., groups of catchments coupled together) were compared to a reference model executed in the standard SWMM configuration. Simulation times increased linearly with the number of connections between components, and mass balance error was a function of the degree to which a component resolved time series data received. This study also examines and proposes some strategies to address these computational costs. We compared coupled OpenMI components with equivalent standard SWMM configuration.SWMM was decomposed by process and space to explore coupling performance penalties.Performance was evaluated in terms of simulation time and mass balance error.We illustrate performance penalties to consider when using component-based models.Strategies to address computational costs of component-based models are proposed.

A data management and publication workflow for a large-scale, heterogeneous sensor network.

It is common for hydrology researchers to collect data using in situ sensors at high frequencies, for extended durations, and with spatial distributions that produce data volumes requiring infrastructure for data storage, management, and sharing. The availability and utility of these data in addressing scientific questions related to water availability, water quality, and natural disasters relies on effective cyberinfrastructure that facilitates transformation of raw sensor data into usable data products. It also depends on the ability of researchers to share and access the data in useable formats. In this paper, we describe a data management and publication workflow and software tools for research groups and sites conducting long-term monitoring using in situ sensors. Functionality includes the ability to track monitoring equipment inventory and events related to field maintenance. Linking this information to the observational data is imperative in ensuring the quality of sensor-based data products. We present these tools in the context of a case study for the innovative Urban Transitions and Aridregion Hydrosustainability (iUTAH) sensor network. The iUTAH monitoring network includes sensors at aquatic and terrestrial sites for continuous monitoring of common meteorological variables, snow accumulation and melt, soil moisture, surface water flow, and surface water quality. We present the overall workflow we have developed for effectively transferring data from field monitoring sites to ultimate end-users and describe the software tools we have deployed for storing, managing, and sharing the sensor data. These tools are all open source and available for others to use.