Michael Cammert

A Cost-Based Approach to Adaptive Resource Management in Data Stream Systems

Data stream management systems need to adaptively control their resources, since stream characteristics and query workload may vary over time. In this paper, we investigate an approach to adaptive resource management for continuous sliding-window queries that adjusts window sizes and time granularities to keep resource usage within bounds. These two novel techniques differ from standard load shedding approaches based on sampling, as they ensure exact query answers for given user-defined quality of service specifications, even under query reoptimization. In order to quantify the effects of both techniques on the various operations in a query plan, we develop an appropriate cost model for estimating operator resource allocation in terms of memory usage and processing costs. A thorough experimental study not only validates the accuracy of our cost model but also demonstrates the efficacy and scalability of the proposed techniques.

Stream Processing in Production-to-Business Software

In order to support continuous queries over data streams, a plethora of suitable techniques as well as prototypes have been developed and evaluated in recent years. In particular, it is of utmost importance to confirm their necessity and feasibility in real-world applications. For that reason, we have successfully coupled our infrastructure for data stream processing (PIPES) with an industrial Production-to-Business software (i-Plant) dedicated to highly automated manufacturing processes.

An Approach to Adaptive Memory Management in Data Stream Systems

Adaptivity is a challenging open issue in data stream management. In this paper, we tackle the problem of memory adaptivity inside a system executing temporal sliding window queries over continuous data streams. Two different techniques to control the memory usage at runtime are proposed which refer to changes in window sizes and time granularities. Both techniques differ from standard load shedding approaches based on sampling as they ensure precise query answers for user-defined Quality of Service (QoS) specifications, even under query re-optimization.

Dynamic plan migration for snapshot-equivalent continuous queries in data stream systems

A data stream management system executes a large number of continuous queries in parallel. As stream characteristics and query workload change over time, the plan initially installed for a continuous query may become inefficient. As a consequence, the query optimizer will re-optimize this plan based on the current statistics. The replacement of the running plan with a more efficient but semantically equivalent plan at runtime is called dynamic plan migration. In order to have a sound semantic foundation for query optimization, we investigate dynamic plan migration for snapshot-equivalent plans. We develop a general method for dynamic plan migration that treats the old and new plan as snapshot-equivalent black boxes. This enables the query optimizer to apply the conventional transformation rules during re-optimization. As a consequence, our approach supports the dynamic optimization of arbitrary continuous queries expressible in CQL, whereas existing solutions are limited in their scope.

Flexible Multi-Threaded Scheduling for Continuous Queries over Data Streams

A variety of real-world applications share the property that data arrives inform of transient streams. Data stream management systems (DSMS) provide convenient solutions to the problem of processing continuous queries on those streams. Within a DSMS, the scheduling of the queries and their operators has proved to be of utmost importance. Previous approaches addressing this issue can be divided into two categories: either each operator runs in its own thread or all operators, combined in one query graph, run in a single thread. Both approaches suffer from severe drawbacks concerning the thread overhead on the one hand and the stalls due to expensive operators on the other hand. To overcome these drawbacks, we propose in this work a hybrid approach that flexibly assigns threads to subgraphs of the query graph. We complement this approach with a suitable strategy to determine these subgraphs. The results of an experimental study substantiate the feasibility of our approach and its superiority to previous approaches.