Leonardo Querzoni

Adaptive online scheduling in storm

Today we are witnessing a dramatic shift toward a data-driven economy, where the ability to efficiently and timely analyze huge amounts of data marks the difference between industrial success stories and catastrophic failures. In this scenario Storm, an open source distributed realtime computation system, represents a disruptive technology that is quickly gaining the favor of big players like Twitter and Groupon. A Storm application is modeled as a topology, i.e. a graph where nodes are operators and edges represent data flows among such operators. A key aspect in tuning Storm performance lies in the strategy used to deploy a topology, i.e. how Storm schedules the execution of each topology component on the available computing infrastructure. In this paper we propose two advanced generic schedulers for Storm that provide improved performance for a wide range of application topologies. The first scheduler works offline by analyzing the topology structure and adapting the deployment to it; the second scheduler enhance the previous approach by continuously monitoring system performance and rescheduling the deployment at run-time to improve overall performance. Experimental results show that these algorithms can produce schedules that achieve significantly better performances compared to those produced by Storms default scheduler.

TERA: topic-based event routing for peer-to-peer architectures

The completely decoupled interaction model offered by the publish/subscribe communication paradigm perfectly suits the interoperability needs of todays large-scale, dynamic, peer-to-peer applications. The unmanaged environments, where these applications are expected to work, pose a series of problems (potentially wide number of partipants, low-reliability of nodes, absence of a centralized authority, etc.) that severely limit the scalability of existing approaches which were originally thought for supporting distributed applications built on the top of static and managed environments. In this paper we propose an architecture for implementing the topic-based publish/subscribe paradigm in large scale peer-to-peer systems. The architecture is based on clustering peers subscribed to the same topic. The major novelty of this architecture lies in the mechanism employed to bring events from the publisher to the cluster (namely outer-cluster routing). The evaluation shows that this mechanism for outer-cluster routing has a probability to bring events to the destination cluster very close to 1 while keeping small the involved number of out-of-cluster peers. Finally, the overall architecture is shown to be scalable along several fundamental dimensions like number of participants, subscriptions, and to exhibit a fair load distribution (load distribution closely follows the distribution of subscriptions on nodes).

Efficient Publish/Subscribe Through a Self-Organizing Broker Overlay and its Application to SIENA

Recently many scalable and efficient solutions for event dissemination in publish/subscribe (pub/sub) systems have appeared in the literature. This dissemination is usually done over an overlay network of brokers and its cost can be measured as the number of messages sent over the overlay to allow the event to reach all intended subscribers. Efficient solutions to this problem are often obtained through smart dissemination algorithms that avoid flooding events on the overlay. In this paper, we propose a complementary approach that obtains efficient event dissemination by reorganizing the overlay network topology. More specifically, this reorganization is done through a self-organizing algorithm executed by brokers whose aim is to directly connect, through overlay links, pairs of brokers matching same events. In this way, on average, the number of brokers involved in an event dissemination decreases, thus reducing its cost. Even though the paradigm of the self-organizing algorithm is general and then applicable to any overlay-based pub/sub system, its concrete implementation depends on the specific system. As a consequence, we studied the effect of the introduction of the self-organizing algorithm in the context of a specific system implementing a tree-based routing strategy, namely SIENA, showing the actual performance benefits through an extensive simulation study. In particular, performance results point out the capacity of the algorithm to converge to an overlay topology accommodating efficient event with respect to (w.r.t) dissemination a specific scenario. Moreover, the algorithm shows a significant capacity to adapt the overlay network topology to continuously changing scenarios while keeping an efficient behavior w.r.t. event dissemination.

Structure-less content-based routing in mobile ad hoc networks

The decoupling and asynchrony properties of the content-based publish-subscribe paradigm makes it very appealing for dynamic wireless networks, like those that often occur in pervasive computing scenarios. Unfortunately, none of the currently available content-based publish-subscribe middleware fit the requirements of such extreme scenarios in which the network is subject to very frequent topological reconfigurations due to the mobility of nodes. In this paper we propose a protocol for content-based message dissemination tailored to mobile ad hoc networks (MANETs) with frequent topological changes. Message routing occurs without the support of any network-wide dispatching infrastructure thus eliminating the issue of maintaining such logical topology on top of a time varying physical topology. The paper reports an extensive simulation study, which provides numerical evidence of the effectiveness of the approach.

A hint-based probabilistic protocol for unicast communications in MANETs ☆

Abstract Point-to-point transmissions represent a fundamental primitive in any communication network. Despite many proposals have appeared in the literature, providing an efficient implementation of such an abstraction in Mobile Ad Hoc Networks (MANETs) still remains an open issue. This paper proposes a probabilistic protocol for unicast packet delivery in a MANET. Unlike the classical routing protocols, in our proposal packet forwarding is not driven by a previously computed path. Rather, the nodes of the network exploit a set of routing meta-information (called hints ) to discover a path to the destination on-the-fly. This assure robustness against topological changes, while requiring a very low overhead. A node gathers hints from the nodes located within a small number of hops (called the protocol’s lookahead) from itself. As showed through simulations, very good performance can be obtained with small lookahead. The main statistical properties of hints have been investigated through an analytical model, which is also reported in the paper.

Content-Based Routing in Highly Dynamic Mobile Ad Hoc Networks

The decoupling and asynchrony properties of the content‐based publish‐subscribe paradigm makes it very appealing for dynamic wireless networks, like those that often occur in pervasive computing scenarios. Unfortunately, most of the currently available content‐based publish‐subscribe middleware do not fit the requirements of such extreme scenarios, in which the network is subject to very frequent topological reconfigurations due to mobility of nodes. In this paper we propose a protocol for content‐based message dissemination tailored to Mobile Ad Hoc Networks (MANETs) showing frequent topological changes. Message routing occurs without the support of any network‐wide dispatching infrastructure thus eliminating the need of maintaining such infrastructure on top of a physical network continuously changing its topology. The paper reports an extensive simulation study that confirms the suitability of the proposed approach along with a stochastic analysis of the central mechanism adopted by the protocol.

Coupling-Based Internal Clock Synchronization for Large-Scale Dynamic Distributed Systems

This paper studies the problem of realizing a common software clock among a large set of nodes without an external time reference (i.e., internal clock synchronization), any centralized control, and where nodes can join and leave the distributed system at their will. The paper proposes an internal clock synchronization algorithm which combines the gossip-based paradigm with a nature-inspired approach, coming from the coupled oscillators phenomenon, to cope with scale and churn. The algorithm works on the top of an overlay network and uses a uniform peer sampling service to fulfill each nodes local view. Therefore, differently from clock synchronization protocols for small scale and static distributed systems, here, each node synchronizes regularly with only the neighbors in its local view and not with the whole system. An evaluation of the convergence speed and the synchronization error of the coupled-based internal clock synchronization algorithm has been carried out, showing how convergence time and the synchronization error depends on the coupling factor and the local view size. Moreover, the variation of the synchronization error with respect to churn and the impact of a sudden variation of the number of nodes have been analyzed to show the stability of the algorithm. In all these contexts, the algorithm shows nice performance and very good self-organizing properties. Finally, we showed how the assumption on the existence of a uniform peer-sampling service is instrumental for the good behavior of the algorithm and how, in system models where network delays are unbounded, a mean-based convergence function reaches a lower synchronization error than median-based convergence functions exploiting the number of averaged clock values.

Efficient key grouping for near-optimal load balancing in stream processing systems

Key grouping is a technique used by stream processing frameworks to simplify the development of parallel stateful operators. Through key grouping a stream of tuples is partitioned in several disjoint sub-streams depending on the values contained in the tuples themselves. Each operator instance target of one sub-stream is guaranteed to receive all the tuples containing a specific key value. A common solution to implement key grouping is through hash functions that, however, are known to cause load imbalances on the target operator instances when the input data stream is characterized by a skewed value distribution. In this paper we present DKG, a novel approach to key grouping that provides near-optimal load distribution for input streams with skewed value distribution. DKG starts from the simple observation that with such inputs the load balance is strongly driven by the most frequent values; it identifies such values and explicitly maps them to sub-streams together with groups of less frequent items to achieve a near-optimal load balance. We provide theoretical approximation bounds for the quality of the mapping derived by DKG and show, through both simulations and a running prototype, its impact on stream processing applications.

Cloud-based data stream processing

In this tutorial we present the results of recent research about the cloud enablement of data streaming systems. We illustrate, based on both industrial as well as academic prototypes, new emerging uses cases and research trends. Specifically, we focus on novel approaches for (1) scalability and (2) fault tolerance in large scale distributed streaming systems. In general, new fault tolerance mechanisms strive to be more robust and at the same time introduce less overhead. Novel load balancing approaches focus on elastic scaling over hundreds of instances based on the data and query workload. Finally, we present open challenges for the next generation of cloud-based data stream processing engines.

Defending financial infrastructures through early warning systems: the intelligence cloud approach

Recent evidence of successful Internet-based attacks and frauds involving financial institutions highlights the inadequacy of the existing protection mechanisms, in which each instutition implements its own isolated monitoring and reaction strategy. Analyzing on-line activity and detecting attacks on a large scale is an open issue due to the huge amounts of events that should be collected and processed. In this paper, we propose a large-scale distributed event processing system, called intelligence cloud, allowing the financial entities to participate in a widely distributed monitoring and detection effort through the exchange and processing of information locally available at each participating site. We expect this approach to be able to handle large amounts of events arriving at high rates from multiple domains of the financial scenario. We describe a framework based on the intelligence cloud where each participant can receive early alerts enabling them to deploy proactive countermeasures and mitigation strategies.