Nithya N. Vijayakumar

Towards low overhead provenance tracking in near real-time stream filtering

Data streams flowing from the physical environment are as unpredictable as the environment itself. Radars go down, long haul networks drop packets, and readings are corrupted on the wire. Yet the data driven scientific models and data mining algorithms do not necessarily account for the inaccuracies when assimilating the data. Low overhead provenance collection partially solves this problem. We propose a data model and collection model for near real time provenance collection. We define a system architecture for stream provenance tracking and motivate with a real-world application in meteorology forecasting.

Real-time storm detection and weather forecast activation through data mining and events processing

Each year across the USA, destructive weather events disrupt transportation and commerce, resulting in both loss of lives and property. Mitigating the impacts of such severe events requires innovative new software tools and cyberinfrastructure through which scientists can monitor data for specific severe weather events such as thunderstorms and launch focused modeling computations for prediction and forecasts of these evolving weather events. Bringing about a paradigm shift in meteorology research and education through advances in cyberinfrastructure is one of the key research objectives of the Linked Environments for Atmospheric Discovery (LEAD) project, a large-scale, interdisciplinary NSF funded project spanning ten institutions. In this paper we address the challenges of making cyberinfrastructure frameworks responsive to real-time conditions in the physical environment driven by the use cases in mesoscale meteorology. The contribution of the research is two-fold: on the cyberinfrastructure side, we propose a model for bridging between the physical environment and e-Science1 workflow systems, specifically through events processing systems, and provide a proof of concept implementation of that model in the context of the LEAD cyberinfrastructure. On the algorithmic side, we propose efficient stream mining algorithms that can be carried out on a continuous basis in real time over large volumes of observational data.

Stream processing in data-driven computational science

The use of real-time data streams in data-driven computational science is driving the need for stream processing tools that work within the architectural framework of the larger application. Data stream processing systems are beginning to emerge in the commercial space, but these systems fail to address the needs of large-scale scientific applications. In this paper we illustrate the unique needs of large-scale data driven computational science through an example taken from weather prediction and forecasting. We apply a realistic workload from this application against our Calder stream processing system to determine effective throughput, event processing latency, data access scalability, and deployment latency

Data Management in Dynamic Environment-driven Computational Science

Advances in numerical modeling, computational hardware and problem solving environments have driven the growth of computational science over the past decades. Science gateways, based on service oriented architectures and scientific workflows, provide yet another step in democratizing access to advanced numerical and scientific tools, computational resource and massive data storage, and fostering collaborations. Dynamic, data-driven applications, such as those found in weather forecasting, present interesting challenges to Science Gateways, which are being addressed as part of the LEAD Cyberinfrastructure project. In this article, we discuss three important data related problems faced by such adaptive data-driven environments: managing a user’s personal workspace and metadata on the Grid, tracking the provenance of scientific workflows and data products, and continuous data mining over observational weather data.

The Neutron Science TeraGrid Gateway: a TeraGrid science gateway to support the Spallation Neutron Source: Research Articles

Web portals are one of the possible ways to access the remote computing resources offered by Grid environments. Since the emergence of the first middleware for the Grid, works have been conducted on delivering the functionality of Grid services on the Web. Many interesting Grid portal solutions have been designed help organize remote access to Grid resources and applications from within Web browsers. They are technically advanced and more and more widely used around the world, resulting in feedback from the community. Some of these user comments concern the flexibility and user-friendliness of the developed solutions. In this paper we present how we addressed the need for a flexible and user-friendly Grid portal environment within the PROGRESS project and how our approach facilitates the use of the Grid within Web portals. Copyright

Evaluation of rate-based adaptivity in asynchronous data stream joins

Continuous query systems are an intuitive way for users to access streaming data in large-scale scientific applications containing many hundreds of streams. A challenge in these systems is to join streams in such a way that memory is conserved. Storing events that could not possibly participate in a join any longer wastes memory and limits scalability of the query processing system. This paper reports an experiment we conducted to validate an algorithm we developed for adaptive rate, adjustable join windows. We posit that a rate-based strategy can result in memory savings, can be sufficiently responsive to rapid changes in stream rates, and can execute with suitably low overhead. Based on the results, we conclude that the algorithm adds between 0.007% and 2.6% overhead, with significant gains in memory utilization possible depending on the particular workload.

Dynamic Filtering and Mining Triggers in Mesoscale Meteorology Forecasting

Mesoscale meteorology forecasting as a data driven application is capable of reacting to events in real-time. We explore a framework for dynamic filtering and mining of data products to generate timely triggers for invoking forecasting applications. In this paper, we present our framework, which couples the Calder stream processing system developed at Indiana University for filter processing and trigger generation, and data mining algorithms developed as part of the ADaM data mining tool kit developed at ITSC, UAH, which detect events for trigger generation.

Calder: enabling grid access to data streams

This paper presents an experimental evaluation of a grid-based continuous query solution to access data streams. The results presented in this study are mixed, however. We are migrating to a netCDF-based streaming model to measure system behavior in a setting that more accurately reflects a real use scenario. We are also working on enabling approximate query processing support to Calder to deal with sudden drop offs and changes in stream rates.

A meta-provenance service to infer context from provenance data of distributed entities

Provenance management has become an integral part of many large-scale distributed computing systems. Tracking the history of data and its usage has led to better understanding of system requirements as well as user needs. Still, the need for an intelligent service that matches the system requirements with user needs is not satisfied. We propose a meta-provenance service that infers context from the provenance information of distributed entities and uses this contextual information to satisfy user needs. We describe our meta-provenance framework by way of describing its implementation in the Calder system. The Calder streaming system enables dynamic invocation of forecast models in LEAD by using a distributed mesh of data mining agents. The meta-provenance service enables sophisticated mapping of user queries from the LEAD portal down to the set of few data mining agents that execute them. Also our meta-provenance service can work at multiple levels of contextual granularity.

Performance evaluation of rate-based join window sizing for asynchronous data streams

Our work is motivated by the large number of data stream sources that define mesoscale meteorology where asynchronous streams are commonplace. Techniques for performing filtering, aggregation, and transformation on multiple streams must be effective for the case of asynchronous streams. Rate Sizing algorithm (RS-Algo) links the number of events waiting to participate in a join to the rate of the streams responsible for their delivery. In this poster, we show the results of performance evaluation of the RS-Algo. The gains in memory utilization are largest under asynchronous streams.