Vincenzo Gulisano

StreamCloud: An Elastic and Scalable Data Streaming System

Many applications in several domains such as telecommunications, network security, large-scale sensor networks, require online processing of continuous data flows. They produce very high loads that requires aggregating the processing capacity of many nodes. Current Stream Processing Engines do not scale with the input load due to single-node bottlenecks. Additionally, they are based on static configurations that lead to either under or overprovisioning. In this paper, we present StreamCloud, a scalable and elastic stream processing engine for processing large data stream volumes. StreamCloud uses a novel parallelization technique that splits queries into subqueries that are allocated to independent sets of nodes in a way that minimizes the distribution overhead. Its elastic protocols exhibit low intrusiveness, enabling effective adjustment of resources to the incoming load. Elasticity is combined with dynamic load balancing to minimize the computational resources used. The paper presents the system design, implementation, and a thorough evaluation of the scalability and elasticity of the fully implemented system.

StreamCloud: A Large Scale Data Streaming System

Data streaming has become an important paradigm for the real-time processing of continuous data flows in domains such as finance, telecommunications, networking, Some applications in these domains require to process massive data flows that current technology is unable to manage, that is, streams that, even for a single query operator, require the capacity of potentially many machines. Research efforts on data streaming have mainly focused on scaling in the number of queries or query operators, but overlooked the scalability issue with respect to the stream volume. In this paper, we present StreamCloud a large scale data streaming system for processing large data stream volumes. We focus on how to parallelize continuous queries to obtain a highly scalable data streaming infrastructure. StreamCloud goes beyond the state of the art by using a novel parallelization technique that splits queries into subqueries that are allocated to independent sets of nodes in a way that minimizes the distribution overhead. StreamCloud is implemented as a middleware and is highly independent of the underlying data streaming engine. We explore and evaluate different strategies to parallelize data streaming and tackle with the main bottlenecks and overheads to achieve scalability. The paper presents the system design, implementation and a thorough evaluation of the scalability of the fully implemented system.

STONE: A streaming DDoS defense framework

Distributed Denial-of-Service (DDoS) attacks aim at rapidly exhausting the communication and computational power of a network target by flooding it with large volumes of malicious traffic. In order to be effective, a DDoS defense mechanism should detect and mitigate threats quickly, while allowing legitimate users access to the attacks target. Nevertheless, defense mechanisms proposed in the literature tend not to address detection and mitigation challenges jointly, but rather focus solely on the detection or the mitigation facet. At the same time, they usually overlook the limitations of centralized defense frameworks that, when deployed physically close to a possible target, become ineffective if DDoS attacks are able to saturate the targets incoming links. This paper presents STONE, a framework with expert system functionality that provides effective and joint DDoS detection and mitigation. STONE characterizes regular network traffic of a service by aggregating it into common prefixes of IP addresses, and detecting attacks when the aggregated traffic deviates from the regular one. Upon detection of an attack, STONE allows traffic from known sources to access the service while discarding suspicious one. STONE relies on the data streaming processing paradigm in order to characterize and detect anomalies in real time. We implemented STONE on top of StreamCloud, an elastic and parallel-distributed stream processing engine. The evaluation, conducted on real network traces, shows that STONE detects DDoS attacks rapidly, provides minimal degradation of legitimate traffic while mitigating a threat, and also exhibits a processing throughput that scales linearly with the number of nodes used to deploy and run it.

Brief announcement: concurrent data structures for efficient streaming aggregation

We briefly describe our study on the problem of streaming multiway aggregation, where large data volumes are received from multiple input streams. Multiway aggregation is a fundamental computational component in data stream management systems, requiring low-latency and high throughput solutions.We focus on the problem of designing concurrent data structures enabling for low-latency and high-throughput multiway aggregation; an issue that has been overlooked in the literature. We propose two new concurrent data structures and their lock-free linearizable implementations, supporting both order-sensitive and order-insensitive aggregate functions.Results from an extensive evaluation show significant improvement in the aggregation performance,in terms of both processing throughput and latency over the commonly-used techniques based on queues.

STONE: a stream-based DDoS defense framework

An effective Distributed Denial of Service (DDoS) defense mechanism must guarantee legitimate users access to an Internet service masking the effects of possible attacks. That is, it must be able to detect threats and discard malicious packets in a online fashion. Given that emerging data streaming technology can enable such mitigation in an effective manner, in this paper we present STONE, a stream-based DDoS defense framework, which integrates anomaly-based DDoS detection and mitigation with scalable data streaming technology. With STONE, the traffic of potential targets is analyzed via continuous data streaming queries maintaining information used for both attack detection and mitigation. STONE provides minimal degradation of legitimate users traffic during DDoS attacks and it also faces effectively flash crowds. Our preliminary evaluation based on an implemented prototype and conducted with real legitimate and malicious traffic traces shows that STONE is able to provide fast detection and precise mitigation of DDoS attacks leveraging scalable data streaming technology.

ScaleJoin: a Deterministic, Disjoint-Parallel and Skew-Resilient Stream Join

The inherently large and varying volumes of information generated in large scale systems demand near real-time processing of data streams. In this context, data streaming is imperative for data-intensive processing infrastructures. Stream joins, the streaming counterpart of database joins, compare tuples coming from different streams and constitute one of the most important and expensive data streaming operators. Algorithmic implementations of stream joins have to be capable of efficiently processing bursty and rate-varying data streams in a deterministic and skew-resilient fashion. To leverage the design of modern multicore architectures, scalability and parallelism need to be addressed also in the algorithmic design. In this paper we present ScaleJoin, an algorithmic construction for deterministic and parallel stream joins that guarantees all the above properties, thus filling in a gap in the existing state-of-theart. Key to the novelty of ScaleJoin is the ScaleGate data structure and its lock-free implementation. ScaleGate facilitates concurrent data exchange and balances independent actions among processing threads; enabling fine-grain parallelism and deterministic processing. It allows ScaleJoin to run on an arbitrary number of processing threads, evenly sharing the overall comparisons run in parallel and achieving disjoint and skew-resilient high processing throughput and low processing latency.

METIS: a Two-Tier Intrusion Detection System for Advanced Metering Infrastructures

In the shift from traditional to cyber-physical electric grids, motivated by the needs for improved energy efficiency, Advanced Metering Infrastructures have a key role. However, together with the enabled possibilities, they imply an increased threat surface on the systems. Challenging aspects such as scalable traffic analysis, timely detection of malicious activity and intuitive ways of specifying detection mechanisms for possible adversary goals are among the core problems in this domain. Aiming at addressing the above, we present METIS, a two-tier streaming-based intrusion detection framework. METIS relies on probabilistic models for detection and is designed to detect challenging attacks in which adversaries aim at being unnoticed. Thanks to its two-tier architecture, it eases the modeling of possible adversary goals and allows for a fully distributed and parallel traffic analysis through the data streaming processing paradigm. At the same time, it allows for complementary intrusion detection systems to be integrated in the framework. We demonstrate METIS’ use and functionality through an energy exfiltration use-case, in which an adversary aims at stealing energy information from AMI users. Based on a prototype implementation using the Storm Stream Processing Engine and a very large dataset from a real-world AMI, we show that METIS is not only able to detect such attacks, but that it can also handle large volumes of data even when run on commodity hardware.

METIS: a two-tier intrusion detection system for advanced metering infrastructures

Specification-based intrusion detection systems, the main defense mechanism proposed so far for Advanced Metering Infrastructures, do not provide a comprehensive protection against the wide spectrum of possible attack scenarios. Challenging aspects in this context include the need for timely detection and for novel attack scenario modeling techniques. This paper introduces METIS, a novel two-tier anomaly-based intrusion detection framework that targets such challenges. The framework provides a continuous and fully distributed processing of network traffic by relying on the data streaming processing paradigm. Attack scenarios can be specified by means of the traffic features they affect and their resulting patterns of malicious activities. We overview the framework, presenting the novel detection technique, and provide results from a case study.

BES: Differentially Private and Distributed Event Aggregation in Advanced Metering Infrastructures

Significant challenges for online event aggregation in the context of Cyber-Physical Systems stem from the computational requirements of their distributed nature, as well as from their privacy concerns. In the context of the latter, differential privacy has gained popularity because of its strong privacy protection guarantees, holding against very powerful adversaries. Despite such strong guarantees, though, its adoption in real-world applications is limited by the privacy-preserving noise it introduces to the analysis, which might compromise its usefulness. We investigate the above problem from a system-perspective in the context of Advanced Metering Infrastructures, providing strong privacy guarantees together with useful results for event aggregation taking into account the distributed nature of such systems. We present a streaming-based framework, Bes, and propose methods to limit the noise introduced by differential privacy in real-world scenarios, thus reducing the resulting utility degradation, while still holding against the adversary model adhering with the original definition of differential privacy. We provide a thorough evaluation based on a fully implemented Bes prototype and conducted with real energy consumption data. We show how a large number of events can be aggregated in a private fashion with low processing latency by a single-board device, similar in performance to the devices deployed in Advanced Metering Infrastructures.

Deterministic real-time analytics of geospatial data streams through ScaleGate objects

This paper presents our solution to the DEBS 2015 Grand Challenge. The analysis of the Grand Challenge is partitioned among an arbitrary number of processing units by leveraging ScaleGate, a recently proposed abstract data type with its concurrent implementation which articulates data access in parallel data streaming. ScaleGate aims not only at supporting high throughput and low latency parallel streaming analysis, but also at guaranteeing deterministic processing, which is one of the biggest challenges in parallelizing computation while maintaining consistency.