Christopher Mutschler

Distributed Low-Latency Out-of-Order Event Processing for High Data Rate Sensor Streams

Event-based Systems (EBS) are used to detect and analyze meaningful events in surveillance, sports, finances and many other areas. With rising data and event rates and with correlations among these events, sequential event processing becomes infeasible and needs to be distributed. Existing approaches cannot deal with the ubiquity of out-of-order event arrival that is introduced by network delays when distributing EBS. Order-less event processing may result in a system failure. We present a low-latency approach based on K-slack that achieves ordered event processing on high data rate sensor and event streams without a-priori knowledge. Slack buffers are dynamically adjusted to fit the disorder in the streams without using local or global clocks. The middleware transparently reorders the event input streams so that events can still be aggregated and processed to a granularity that satisfies the demands of the application. On a Realtime Locating System (RTLS) our system performs accurate low-latency event detection under the predominance of out-of-order event arrival and with a close to linear performance scale-up when the system is distributed over several threads and machines.

Reliable speculative processing of out-of-order event streams in generic publish/subscribe middlewares

In surveillance, sports, finances, etc., distributed event-based systems are used to detect meaningful events with low latency in high data rate event streams. Both known approaches to deal with the predominant out-of-order event arrival at the distributed detectors have their shortcomings: buffering approaches introduce latencies for event ordering and stream revision approaches may result in system overloads due to unbounded retraction cascades. This paper presents a speculative processing technique for out-of-order event streams that enhances typical buffering approaches. In contrast to other stream revision approaches our novel technique encapsulates the event detector, uses the buffering technique to delay events but also speculatively processes a portion of it, and adapts the degree of speculation at runtime to fit the available system resources so that detection latency becomes minimal. Our technique outperforms known approaches on both synthetical data and real sensor data from a Realtime Locating System (RTLS) with several thousands of out-of-order sensor events per second. Speculative buffering exploits system resources and reduces latency by 40% on average.

Learning event detection rules with noise hidden Markov models

Complex Event Processing (CEP) is a popular method to monitor processes in several contexts, especially when dealing with incidents at distinct points in time. Specific temporal combinations of various events are often of special interest for automatic detection. For the description of such patterns, one can either implement rules in some higher programming language or use some Event Description Language (EDL). Both is complicated and error-prone for non-engineers, because it varies greatly from natural language. Therefore, we present a method, by which a domain expert can simply signal the occurrence of a significant incident at a specific point in time. The system then infers rules for automatically detecting such occurrences later on. At the core of our approach is an extension of hidden Markov models (HMM) called noise hidden Markov models (nHMM) that can be trained with existing, low-level event data. The nHMM can be applied online without any intervention of programming experts. An evaluation on both synthetic and real event data shows the efficiency of our approach even under the presence of highly frequent, insignificant events and uncertainty in the data.

Predictive load management in smart grid environments

The DEBS 2014 Grand Challenge targets the monitoring and prediction of energy loads of smart plugs installed in private households. This paper presents details of our middleware solution and efficient median calculation, shows how we address data quality issues, and provides insights into our enhanced prediction based on hidden Markov models. The evaluation on the smart grid data set shows that we process up to 244k input events per second with an average detection latency of only 13.3ms, and that our system efficiently scales across nodes to increase throughput. Our prediction model significantly outperforms the median-based prediction as it deviates much less from the real load values, and as it consumes considerably less memory.

Adaptive Speculative Processing of Out-of-Order Event Streams

Distributed event-based systems are used to detect meaningful events with low latency in high data-rate event streams that occur in surveillance, sports, finances, etc. However, both known approaches to dealing with the predominant out-of-order event arrival at the distributed detectors have their shortcomings: buffering approaches introduce latencies for event ordering, and stream revision approaches may result in system overloads due to unbounded retraction cascades. This article presents an adaptive speculative processing technique for out-of-order event streams that enhances typical buffering approaches. In contrast to other stream revision approaches developed so far, our novel technique encapsulates the event detector, uses the buffering technique to delay events but also speculatively processes a portion of it, and adapts the degree of speculation at runtime to fit the available system resources so that detection latency becomes minimal. Our technique outperforms known approaches on both synthetical data and real sensor data from a realtime locating system (RTLS) with several thousands of out-of-order sensor events per second. Speculative buffering exploits system resources and reduces latency by 40% on average.

Runtime migration of stateful event detectors with low-latency ordering constraints

Runtime migration has been widely adopted to achieve several tasks such as load balancing, performance optimization, and fault-tolerance. However, existing migration techniques do not work for event detectors in distributed publish/subscribe systems that are used to analyze sensor data. Since low-latency time-constraints are no longer valid they reorder streams incorrectly and cause erroneous event detector states. This paper presents a safe runtime migration of stateful event detectors that respects low-latency time-constraints and seamlessly orders input events correctly on the migrated host. Event streams are only forwarded until timing delays are properly calibrated, the migrated event detector immediately stops processing after its state is transferred, and the processing overhead is negligible. On a Realtime Locating System (RTLS) we show that we can efficiently migrate event detectors at runtime between servers where other techniques would fail.

[POSTER] Social Augmentations in Multi-User Virtual Reality: A Virtual Museum Experience

This work in progress report demonstrates a novel approach for behavioral augmentations in Virtual Reality (VR). Using a large scale tracking system, groups of five users explored a virtual museum. We investigated how augmenting social interactions impacts this experience, by designing behavioral transformations for behavioral phenomena in social interactions. Preliminary data indicate a reduction of perceived isolation, and a more thought-provoking experience with active behavioral augmentations.

Low-complexity PDoA-based localization

Localization of wireless nodes within the IoT received much attention lately. However, strong constraints on power consumption, scalability, and complexity of the nodes pose a big challenge for localization techniques. This paper presents a concept for energy-efficient low-complexity localization based on Phase Difference of Arrival (PDoA). Besides a novel method for reference transmitter selection we propose a waveform, well-suited for PDoA measurements, and evaluate its ranging performance. We compare multiple signal classification (MUSIC), linear fitting, and mean phase difference and compare their estimation variance to the Cramer Rao Lower Bound (CRLB). Our system concept allows for the mitigation of near-far effects for reference and tag signals at the receiver nodes, and an efficient implementation of a wideband frequency hopping scheme.

Demo: do event-based systems have a passion for sports?

The ubiquity of sensor data calls for automatic processing to extract valuable information. Realtime Locating Systems (RTLS) provide many parallel position data streams for interacting objects, and event-based systems are the method of choice to analyze them. We demonstrate a distributed event processing system for position stream data from a Realtime Locating System used for a soccer application. Our system can deal with the insufficient knowledge on object and system behavior, and thus the event data loads at runtime. To do so, it dynamically adapts to the variations in the observed environment: events are ordered with respect to their delays, event detectors are reconfigured and migrated between nodes at runtime, and the system is scalable as the number of trackable objects and sensors changes. We demonstrate the efficiency of our system architecture and provide tools to visualize data and to configure detection units at runtime.

Evolutionary algorithms that use runtime migration of detector processes to reduce latency in event-based systems

Event-based systems (EBS) are widely used to efficiently process massively parallel data streams. In distributed event processing the allocation of event detectors to machines is crucial for both the latency and efficiency, and a naive allocation may even cause a system failure. But since data streams, network traffic, and event loads cannot be predicted sufficiently well the optimal detector allocation cannot be found a-priori and must instead be determined at runtime. This paper describes how evolutionary algorithms (EA) can be used to minimize both network and processing latency by means of runtime migration of event detectors. The paper qualitatively evaluates the algorithms on synthetical data streams in a distributed event-based system. We show that some EAs work efficiently even with large numbers of event detectors and machines and that a hybrid of Cuckoo Search and Particle Swarm Optimization outperforms others.