Nihal Dindar

SECRET: a model for analysis of the execution semantics of stream processing systems

The IEEE Board of Directors is due to consider a proposal to change IEEE Policy Statement 9.7 to permit IEEE entities to sponsor or cosponsor classified sessions anywhere in the world. The policy has been to prohibit such (co)sponsorship on the grounds that every dues-paying IEEE member has a fundamental right to attend any and all IEEE events. That right should not be abrogated.There are many academic and commercial stream processing engines (SPEs) today, each of them with its own execution semantics. This variation may lead to seemingly inexplicable differences in query results. In this paper, we present SECRET, a model of the behavior of SPEs. SECRET is a descriptive model that allows users to analyze the behavior of systems and understand the results of window-based queries for a broad range of heterogeneous SPEs. The model is the result of extensive analysis and experimentation with several commercial and academic engines. In the paper, we describe the types of heterogeneity found in existing engines, and show with experiments on real systems that our model can explain the key differences in windowing behavior.

DejaVu: declarative pattern matching over live and archived streams of events

DejaVu is an event processing system that integrates declarative pattern matching over live and archived streams of events on top of a novel system architecture. We propose to demonstrate the key aspects of the DejaVu query language and architecture using two different application scenarios, namely a smart RFID library system and a financial market data analysis application. The demonstration will illustrate how DejaVu can uniformly handle one-time, continuous, and hybrid pattern matching queries over live and archived stream stores, using highly interactive visual monitoring tools including one that is based on the Second Life virtual world.

Efficiently correlating complex events over live and archived data streams

Correlating complex events over live and archived data streams, which we call Pattern Correlation Queries (PCQs), provides many benefits for domains which need real time forecasting of events or identification of causal dependencies, while handling data at high rates and in massive amounts, like in financial or medical settings. Existing work has focused either on complex event processing over a single type of stream source (i.e., either live or archived), or on simple stream correlation queries (e.g., live events trigerring a database lookup). In this paper, we specifically focus on recency-based PCQs and provide clear, useful, and optimizable semantics for them. PCQs raise a number of challenges in optimizing data management and query processing, which we address in the setting of the DejaVu complex event processing system. More specifically, we propose three complementary optimizations including recent input buffering, query result caching, and join source ordering. Furthermore, we capture the relevant query processing tradeoffs in a cost model. An extensive performance study on synthetic and real-life data sets not only validates this cost model, but also shows that our optimizations are very effective, achieving more than two orders magnitude throughput improvement and much better scalability compared to a conventional approach.

Modeling the execution semantics of stream processing engines with SECRET

There are many academic and commercial stream processing engines (SPEs) today, each of them with its own execution semantics. This variation may lead to seemingly inexplicable differences in query results. In this paper, we present SECRET, a model of the behavior of SPEs. SECRET is a descriptive model that allows users to analyze the behavior of systems and understand the results of window-based queries (with time- and tuple-based windows) for a broad range of heterogeneous SPEs. The model is the result of extensive analysis and experimentation with several commercial and academic engines. In the paper, we describe the types of heterogeneity found in existing engines and show with experiments on real systems that our model can explain the key differences in windowing behavior.

RIP: run-based intra-query parallelism for scalable complex event processing

Recognition of patterns in event streams has become important in many application areas of Complex Event Processing (CEP) including financial markets, electronic health-care systems, and security monitoring systems. In most applications, patterns have to be detected continuously and in real-time over streams that are generated at very high rates, imposing high-performance requirements on the underlying CEP system. For scaling CEP systems to increasing workloads, parallel pattern matching techniques that can exploit multi-core processing opportunities are needed. In this paper, we propose RIP - a Run-based Intra-query Parallelism technique for scalable pattern matching over event streams. RIP distributes input events that belong to individual run instances of a patterns Finite State Machine (FSM) to different processing units, thereby providing fine-grained partitioned data parallelism. We compare RIP to a state-based alternative which partitions individual FSM states to different processing units instead. Our experiments demonstrate that RIPs partitioned parallelism approach outperforms the pipelined parallelism approach of this state-based alternative, achieving near-linear scalability that is independent from the query pattern definition.

Federated Stream Processing Support for Real-Time Business Intelligence Applications

In this paper, we describe the MaxStream federated stream processing architecture to support real-time business intelligence applications. MaxStream builds on and extends the SAP MaxDB relational database system in order to provide a federator over multiple underlying stream processing engines and databases. We show preliminary results on usefulness and performance of the MaxStream architecture on the SAP Sales and Distribution Benchmark.

A demonstration of the MaxStream federated stream processing system

MaxStream is a federated stream processing system that seamlessly integrates multiple autonomous and heterogeneous Stream Processing Engines (SPEs) and databases. In this paper, we propose to demonstrate the key features of MaxStream using two application scenarios, namely the Sales Map & Spikes business monitoring scenario and the Linear Road Benchmark, each with a different set of requirements. More specifically, we will show how the MaxStream Federator can translate and forward the application queries to two different commercial SPEs (Coral8 and StreamBase), as well as how it does so under various persistency requirements.

Event Processing Support for Cross-Reality Environments

Complex event processing (CEP) is an essential functionality for cross-reality environments. Through CEP, we can turn raw sensor data generated in the real world into more meaningful information that has some significance for the virtual world. In this article, the authors present DejaVu, a general-purpose event processing system built at ETH Zurich. SmartRFLib, a cross-reality application, builds on DejaVu and enables real-time event detection over RFID data streams feeding a virtual library on second life.

DejaVu: a complex event processing system for pattern matching over live and historical data streams

This short paper provides an overview of the DejaVu complex event processing (CEP) system, with an emphasis on its novel architecture and query optimization techniques for correlating patterns across live and historical data streams.