Niko Pollner

Efficient and cost-aware operator placement in heterogeneous stream-processing environments

Operator placement for distributed stream-processing systems is still a challenging problem that can be modeled as a Task Assignment Problem (TAP). Multiple objectives are relevant for the optimization in heterogeneous stream-processing systems as there are different capabilities of the underlying networks and stream-processing nodes. We present an approach based on linear programming relaxation and iterative deterministic rounding. It uses an efficient linearization approach for the quadratic objective function that results from the TAP.

Operator fission for load balancing in distributed heterogeneous data stream processing systems

Distributed data stream processing allows to optimize resource consumption. A querys operators can be executed by several systems. The placement of filter or aggregate operators near the data source omits unnecessary data transfer. The operator placement decision is a complex problem. In certain scenarios the goal is not only overall minimization of e.g. resource consumption but an evenly distributed load. We propose an operator fission algorithm, that works on the basis of an initial operator placement. The algorithm selects certain operators from the set of operators that allow fission for parallel execution by multiple systems. Load is thus divided between processors in a more fine-grained way, resulting in lower maximum load and lower load variance. We present and evaluate three different variants of the algorithm to allow tuning the trade-off between optimization time and result quality.

An algebra for pattern matching, time-aware aggregates and partitions on relational data streams

Many interesting applications of continuous-query processing are concerned with pattern matching or complex temporal aggregation of events. Real-world queries that rely on these operations are difficult to implement in current stream-processing systems. The reason seems to be a gap between two types of existing query languages: Some languages (e. g. CQL) offer a small set of simple operators that can be combined in order to create complex queries. While these languages provide sound and comprehensible semantics, they lack the expressiveness required for many real-world applications. Other approaches (e. g. Aurora) provide powerful operators but lack semantic strictness, which is required for reasoning about query results. Such reasoning is a prerequisite for safe query optimization. We try to bridge this gap by integrating operators for pattern matching and time-aware aggregates into a general-purpose stream model featuring stream partitioning. These operators can answer several questions that we have found to be relevant in a real-world object-tracking scenario. Moreover, they are formally defined, allowing expressive and efficient queries to be written in CQL-like languages, while remaining understandable and easy to use.

Black-box determination of cost models' parameters for federated stream-processing systems

For distribution and deployment of queries in distributed stream-processing environments, it is vital to estimate the expected costs in advance. Having heterogeneous Stream-Processing Systems (SPSs) running on various hosts, the parameters of a cost model for an operator must be determined by measurements for each relevant combination of an SPS and hardware. This paper presents a black-box method that determines the parameters of appropriate cost models that regard system-specific behavior. For some SPSs, there might not be any appropriate cost model available due to the lack of internal knowledge. If no cost model is available for any reason, we provide and apply a non-parametric model.

An overlay network for integration of WSNs in federated stream-processing environments

This paper presents an approach for seamless integration of hosts in heterogeneous networks in the context of data-stream processing. The integration of multiple heterogeneous hosts from the sensor-node level up to the level of high-performance workstations is one of the most promising concepts for extensive and efficient analysis of streaming data. For controlling such a network, communication between hosts is needed, e.g., to initiate stream processing, to configure queries, and to transmit streams. One of the key challenges is global addressing and transparent yet efficient data exchange despite diverse, differently capable networks involved. For this purpose, we developed a cross-platform overlay network that enables transparent communication between autonomous stream-processing systems on different hosts in miscellaneous networks for both data streams and control commands. The system directly uses underlying native protocols within each network so that the most efficient communication method is applied. Furthermore, global addressing of instances of stream-processing systems and routing over the individual communication paths is provided.

Placement-Safe Operator-Graph Changes in Distributed Heterogeneous Data Stream Systems

Data stream processing systems enable querying continuous data without first storing it. Data stream queries may combine data from distributed data sources like different sensors in an environmental sensing application. This suggests distributed query processing. Thus the amount of transferred data can be reduced and more processing resources are available.However, distributed query processing on probably heterogeneous platforms complicates query optimization. This article investigates query optimization through operator graph changes and its interaction with operator placement on heterogeneous distributed systems. Pre-placement operator graph changes may prevent certain operator placements. Thereby the resource consumption of the query execution may unexpectedly increase. Based on the operator placement problem modeled as a task assignment problem (TAP), we prove that it is NP-hard to decide in general whether an arbitrary operator graph change may negatively influence the best possible TAP solution. We present conditions for several specific operator graph changes that guarantee to preserve the best possible TAP solution.

Umgang mit semantischer Heterogenität bei der Integration stromverarbeitender Systeme

ZusammenfassungVerteilte Datenstromverarbeitung unter Beteiligung heterogener Datenstromsysteme und Sensorknoten erfährt zunehmendes Interesse. Ein Problem dabei ist, dass die heute verfügbaren Datenstromsysteme sich hinsichtlich ihrer Verarbeitungslogik unterscheiden. Das zeigt sich darin, dass für vermeintlich gleiche Anfragen unterschiedliche Ergebnisse erzeugt werden bzw. Ergebnisströme unterschiedliches zeitliches Verhalten aufweisen. Problematisch ist das insbesondere für die automatische Integration heterogener Datenstromsysteme im Sinne einer Föderation, denn hier gibt der Anwendungsentwickler die Kontrolle darüber, wie und von welchem System Ergebnisse erzeugt werden, an einen Koordinator ab. Der möglichen Abweichungen muss sich der Anwendungsentwickler bewusst sein und er muss angeben können, welche davon er in Kauf nehmen will. Dieser Beitrag beschreibt einen Ansatz, der es dem Anwendungsentwickler wahlweise erlaubt, präzise zu definieren, wie eine Anfrage verarbeitet werden soll oder dem System bestimmte Teilaspekte freizustellen, um so Optimierungspotentiale zu nutzen. Unterstützt wird er dabei durch eine Visualisierung der möglichen Unterschiede im Anfrageergebnis.

Data Stream Application Manager (DSAM)

We present our Data Stream Application Manager (DSAM). DSAM integrates different data stream management systems (DSMS) and also sensor network nodes. The user enters a query written in a system-independent declarative query language into a central control program. DSAM analyzes this query and distributes it to the available DSMSs. The distribution algorithm uses a cost model to minimize overall processing and transmission costs. The demo shows some scenarios in which DSAM can be used. We do not focus on concrete applications but on generic use-cases where DSAM may help users or application developers. One example is switching to a different DSMS without changing queries. Another example is the automatic integration of different DSMSs to make them appear just like one system.