Quinn Hart

Towards Reliable, Performant Workflows for Streaming-Applications on Cloud Platforms

Scientific workflows are commonplace in eScience applications. Yet, the lack of integrated support for data models, including streaming data, structured collections and files, is limiting the ability of workflows to support emerging applications in energy informatics that are stream oriented. This is compounded by the absence of Cloud data services that support reliable and performant streams. In this paper, we propose and present a scientific workflow framework that supports streams as first-class data, and is optimized for performant and reliable execution across desktop and Cloud platforms. The workflow framework features and its empirical evaluation on a private Eucalyptus cloud are presented.

California Simulation of Evapotranspiration of Applied Water and Agricultural Energy Use in California

Abstract The California Simulation of Evapotranspiration of Applied Water (Cal-SIMETAW) model is a new tool developed by the California Department of Water Resources and the University of California, Davis to perform daily soil water balance and determine crop evapotranspiration (ET c ), evapotranspiration of applied water (ET aw ), and applied water (AW) for use in California water resources planning. ET aw is a seasonal estimate of the water needed to irrigate a crop assuming 100% irrigation efficiency. The model accounts for soils, crop coefficients, rooting depths, seepage, etc. that influence crop water balance. It provides spatial soil and climate information and it uses historical crop and land-use category information to provide seasonal water balance estimates by combinations of detailed analysis unit and county (DAU/County) over California. The result is a large data base of ET c and ET aw that will be used to update information in the new California Water Plan (CWP). The application uses the daily climate data, i.e., maximum (T x ) and minimum (T n ) temperature and precipitation (P cp ), which were derived from monthly USDA-NRCS PRISM data (PRISM Group 2011) and daily US National Climate Data Center (NCDC) climate station data to cover California on a 4 km×4 km change grid spacing. The application uses daily weather data to determine reference evapotranspiration (ET o ), using the Hargreaves-Samani (HS) equation (Hargreaves and Samani 1982, 1985). Because the HS equation is based on temperature only, ET o from the HS equation were compared with CIMIS ET o at the same locations using available CIMIS data to determine correction factors to estimate CIMIS ET o from the HS ET o to account for spatial climate differences. Cal-SIMETAW also employs near real-time reference evapotranspiration (ET o ) information from Spatial CIMIS, which is a model that combines weather station data and remote sensing to provide a grid of ET o information. A second database containing the available soil water holding capacity and soil depth information for all of California was also developed from the USDA-NRCS SSURGO database. The Cal-SIMETAW program also has the ability to generate daily weather data from monthly mean values for use in studying climate change scenarios and their possible impacts on water demand in the state. The key objective of this project is to improve the accuracy of water use estimates for the California Water Plan (CWP), which provides a comprehensive report on water supply, demand, and management in California. In this paper, we will discuss the model and how it determines ET aw for use in water resources planning.

A data and query model for streaming geospatial image data

Most of the recent work on adaptive processing and continuous querying of data streams assume that data objects come in the form of tuples, thus relying on the relational data model and traditional relational operators as basis for query processing techniques. Complex types of objects, such as multidimensional data sets or the vast amounts of raster image data continuously streaming down to Earth from satellites have not been considered. In this paper, we introduce a data and query model as a comprehensive and practically relevant basis for managing and querying streams of remotely-sensed geospatial image data. Borrowing basic concepts from Image Algebra, we detail a data model that reflects basic properties of such streams of imagery. We present a query model that includes stream restrictions, transforms, and compositions, and provides a sound basis for formulating expressive and practically relevant queries over streams of image data. Finally, we outline how the data and query model is currently realized in a data stream management system for geospatial image data that supports geographic applications.

Streaming satellite data to cloud workflows for on-demand computing of environmental data products

Environmental data arriving constantly from satellites and weather stations are used to compute weather coefficients that are essential for agriculture and viticulture. For example, the reference evapotranspiration (ET0) coefficient, overlaid on regional maps, is provided each day by the California Department of Water Resources to local farmers and turf managers to plan daily water use. Scaling out single-processor compute/data intensive applications operating on realtime data to support more users and higher-resolution data poses data engineering challenges. Cloud computing helps data providers expand resource capacity to meet growing needs besides supporting scientific needs like reprocessing historic data using new models. In this article, we examine migration of a legacy script used for daily ET0 computation by CIMIS to a workflow model that eases deployment to and scaling on the Windows Azure Cloud. Our architecture incorporates a direct streaming model into Cloud virtual machines (VMs) that improves the performance by 130% to 160% for our workflow over using Cloud storage for data staging, used commonly. The streaming workflows achieve runtimes comparable to desktop execution for single VMs and a linear speed-up when using multiple VMs, thus allowing computation of environmental coefficients at a much larger resolution than done presently.

Optimization of multiple continuous queries over streaming satellite data

Remotely sensed data, in particular satellite imagery, play many important roles in environmental applications. In particular applications that study rapid changes in the environment require frequent access to these data. For continuous data products, users are often interested in formulating continuous queries that deliver results for each incoming image. In the presence of multiple continuous queries, there is clearly an opportunity to share common intermediate data and thus, increase the overall processing speed of the system.Based on the widely used GRASS, this paper describes a system that realizes multiple query processing using two major components. A query optimizer maintains the current set of active continuous queries. Queries are organized into a single processing plan designed to share intermediate results. For each new image from the stream, the optimizer generates an execution plan specific to the active queries. The query executor then rewrites this plan into a set of geospatial processing steps and executes the plan.We detail experiments using data from NOAAs GOES. Continuous queries are defined in a way similar to the OGC WMS query specification. Using predicted query patterns over the visible hemisphere of GOES, experimental results indicate that multiple-query optimized plans can improve performance significantly when compared to queries that are executed separately.

Sensor data dissemination systems using Web-based standards: a case study of publishing data in support of evapotranspiration models in California

Developing real-time and near real-time systems and models based on sensor data acquisition requires efficient access to distributed data sources. This is especially true when multi-source datasets are required for a comprehensive model. For example, developing daily spatial estimations of water evaporation and transpiration requires access to weather stations, combined with satellite imagery, and potentially weather prediction models as well. Most existing online data dissemination services support dissemination with ad-hoc methods, designed for specific limited purposes. General purpose access methods, required by a broadening range of uses, become increasingly important. In this study, we examine standard Web services for sensor data and how clients can consume such services seamlessly in their applications. Specifically, we describe a prototype system that supports Open Geospatial Consortium standards and the Network Data Access Protocol standard. This system allows standard access to weather stations, which provide comprehensive, timely weather data, as well as access to NOAA GOES satellite data for real-time remote sensing images of California. The prototype provides daily reference evapotranspiration map for California. In addition to presenting the technical details of the implementation, the costs and benefits of building such a standard-based sensor data dissemination system are discussed.

Evaluation of a dynamic tree structure for indexing query regions on streaming geospatial data

Most recent research on querying and managing data streams has concentrated on traditional data models where the data come in the form of tuples or XML data. Complex types of streaming data, in particular spatio-temporal data, have primarily been investigated in the context of moving objects and location-aware services. In this paper, we study query processing and optimization aspects for streaming (RSI) data. Streaming RSI is typical for the vast amount of imaging satellites orbiting the Earth, and it exhibits certain characteristics that make it very attractive to tailored query optimization techniques. Our approach uses a Dynamic Cascade Tree (DCT) to (1) index spatio-temporal query regions associated with continuous user queries and (2) efficiently determine what incoming RSI data is relevant to what queries. The (DCT) supports the processing of different types of RSI data, ranging from point data to more general spatial extents in which the incoming imagery can be single pixels, rows of pixels, or discrete parts of images. The DCT exploits spatial trends in incoming RSI data to efficiently filter the data of interest to the individual query regions. Experimental results using random input and Geostationary Operational Environmental Satellite (GOES) data give a good insight into processing streaming RSI and verify the efficiency and utility of the DCT .

Design of a GIS-Based Web Application for Simulating Biofuel Feedstock Yields

Short rotation woody crops (SRWC), such as hybrid poplar, have the potential to serve as a valuable feedstock for cellulosic biofuels. Spatial estimates of biomass yields under different management regimes are required for assisting stakeholders in making better management decisions and to establish viable woody cropping systems for biofuel production. To support stakeholders in their management decisions, we have developed a GIS-based web interface using a modified 3PG model for spatially predicting poplar biomass yields under different management and climate conditions in the U.S. Pacific Northwest region. The application is implemented with standard HTML5 components, allowing its use in a modern browser and dynamically adjusting to the client screen size and device. In addition, cloud storage of the results makes them accessible on any Internet-enabled device. The web interface appears simple, but is powerful in parameter manipulation and in visualizing and sharing the results. Overall, this application comprises dynamic features that enable users to run SRWC crop growth simulations based on GIS information and contributes significantly to choosing appropriate feedstock growing locations, anticipating the desired physiological properties of the feedstock and incorporating the management and policy analysis needed for growing hybrid poplar plantations.

Sliding Window Calculations on Streaming Data using the Kepler Scientific Workflow System

Abstract In many areas of science unbounded (potentially infinite) data streams need to be processed in a continuous manner, e.g., to compute running aggregates or sliding window aggregates. One important example is the computation of Growing Degree Days (GDD) from a stream of temperature data, which provides a heuristic tool to predict plant development and the maturity of crops. The process of data acquisition, processing, storage, and presentation forms a scientific workflow and scientific workflow systems have been developed to automate their execution. The whole workflow is decomposed into its individual steps, represented by actors , which in turn are connected by channels that describe the flow of data. This workflow representation allows to reuse existing components for different workflows, and, in principle, easy modification of existing workflows. In current streaming workflow designs in Kepler, data belonging to a particular time window is typically identified by counting data tokens on channels between actors. For example, this token-counting approach does not work for windows of variable length nor for overlapping windows. In this paper, we address these limitations and present a new actor design with two incoming streams: a time-stamp ordered data stream, and a stream of aggregation windows, ordered by their start time. We present a new Chunker actor that “stream-joins” the data from one stream with the windows presented on the second stream, where windows represent aggregation intervals of variable length and possibly overlapping time. Windows containing the corresponding data are output as soon as they are completed, i.e. once timestamps in the data stream pass the end time of a window. We illustrate the approach with an improved GDD workflow based on our new Chunker actor.

Hybrid Poplar based Biorefinery Siting Web Application (HP-BiSWA): An online decision support application for siting hybrid poplar based biorefineries

Abstract Hybrid poplar has potential as feedstock for the production of bioenergy and bio-based products. Planning effective placement of hybrid poplar based biorefinery facilities takes tremendous time and financial resources. A hybrid poplar based biorefinery siting application (HP-BiSWA), a web-enabled refinery siting application, was developed so that potential stakeholders may quickly assess available resources at selected locations and provide information relating to the economic competitiveness and financial risks associated with construction and operation. At present, the tool supports evaluation of the potential for hybrid poplar based jet fuel and acetic acid production based on user-specified conditions. The application is generally expandable to other feedstock, technology, and product types. HP-BiSWA uses various modules (i.e. 3PG-crop growth model and farm budget application) and services (i.e. parcel service, transportation routing service, crop service, soil and weather services) to retrieve and estimate information to optimize and select potential parcels for poplar cultivation, and ultimately determine net revenue for biofuel production under selected decision options. To demonstrate the utility of HP-BiSWA, a case study analysis was performed for Centralia, WA based on a 380 ML/yr (100 MGY) jet fuel biorefinery using two scenarios: (1) a 175 km feedstock supply radius with 50% pastureland use, and (2) a 225 km feedstock supply radius with 25% pastureland use. The case study captures the interactive effects on biorefinery performance with changes in feedstock supply area and available water, power, and other resources for operation of the biorefinery.