Yann Busnel

Efficient key grouping for near-optimal load balancing in stream processing systems

Key grouping is a technique used by stream processing frameworks to simplify the development of parallel stateful operators. Through key grouping a stream of tuples is partitioned in several disjoint sub-streams depending on the values contained in the tuples themselves. Each operator instance target of one sub-stream is guaranteed to receive all the tuples containing a specific key value. A common solution to implement key grouping is through hash functions that, however, are known to cause load imbalances on the target operator instances when the input data stream is characterized by a skewed value distribution. In this paper we present DKG, a novel approach to key grouping that provides near-optimal load distribution for input streams with skewed value distribution. DKG starts from the simple observation that with such inputs the load balance is strongly driven by the most frequent values; it identifies such values and explicitly maps them to sub-streams together with groups of less frequent items to achieve a near-optimal load balance. We provide theoretical approximation bounds for the quality of the mapping derived by DKG and show, through both simulations and a running prototype, its impact on stream processing applications.

Uniform and ergodic sampling in unstructured peer-to-peer systems with malicious nodes

We consider the problem of uniform sampling in large scale open systems. Uniform sampling is a fundamental primitive that guarantees that any individual in a population has the same probability to be selected as sample. An important issue that seriously hampers the feasibility of uniform sampling in open and large scale systems is the unavoidable presence of malicious nodes. In this paper we show that restricting the number of requests that malicious nodes can issue and allowing for a full knowledge of the composition of the system is a necessary and sufficient condition to guarantee uniform and ergodic sampling. In a nutshell, a uniform and ergodic sampling guarantees that any node in the system is equally likely to appear as a sample at any non malicious node in the system and that infinitely often any node has a nonnull probability to appear as a sample of honest nodes.

Analysis of Deterministic Tracking of Multiple Objects Using a Binary Sensor Network

Let consider a set of anonymous moving objects to be tracked in a binary sensor network. This article studies the problem of associating deterministically a track revealed by the sensor network with the trajectory of an unique anonymous object, namely the multiple object tracking and identification (MOTI) problem. In our model, the network is represented by a sparse connected graph where each vertex represents a binary sensor and there is an edge between two sensors if an object can pass from one sensed region to another one without activating any other sensor. The difficulty of MOTI lies in the fact that the trajectories of two or more objects can be so close that the corresponding tracks on the sensor network can no longer be distinguished (track merging), thus confusing the deterministic association between an object trajectory and a track. The article presents several results. We first show that MOTI cannot be solved on a general graph of ideal binary sensors even by an omniscient external observer if all the objects can freely move on the graph. Then we describe restrictions that can be imposed a priori either on the graph, on the object movements, or on both, to make the MOTI problem always solvable. In the absence of an omniscient observer, we show how our results can lead to the definition of distributed algorithms that are able to detect when the system is in a state where MOTI becomes unsolvable.

Gossiping over storage systems is practical

Gossip-based mechanisms are touted for their simplicity, limited resource usage, robustness to failures, and tunable system behavior. These qualities make gossiping an ideal mechanism for storage systems that are responsible for maintaining and updating data in a mist of failures and limited resources (e.g., intermittent network connectivity, limited bandwidth, constrained communication range, or limited battery power). We focus on persistent storage systems that, unlike mere caches, are responsible for both the durability and the consistency of data. Examples of such systems may be encountered in many different environments, in particular: wide-area networks (constrained by limited bandwidth), wireless sensor networks (characterized by limited resources), and mobile ad hoc networks (suffering from intermittent connectivity). In this paper, we demonstrate the qualities of gossiping in these three respective environments.

Efficiently Summarizing Data Streams over Sliding Windows

Estimating the frequency of any piece of information in large-scale distributed data streams became of utmost importance in the last decade (e.g., in the context of network monitoring, big data, etc.). If some elegant solutions have been proposed recently, their approximation is computed from the inception of the stream. In a runtime distributed context, one would prefer to gather information only about the recent past. This may be led by the need to save resources or by the fact that recent information is more relevant. In this paper, we consider the sliding window model and propose two different (on-line) algorithms that approximate the items frequency in the active window. More precisely, we determine a (ε, δ)-additive-approximation meaning that the error is greater than ε only with probability δ. These solutions use a very small amount of memory with respect to the size N of the window and the number n of distinct items of the stream, namely, O(1/ε log 1/δ (log N+log n)) and O(1/τε log 1/δ (log N+log n)) bits of space, where τ is a parameter limiting memory usage. We also provide their distributed variant, i.e., considering the sliding window functional monitoring model. We compared the proposed algorithms to each other and also to the state of the art through extensive experiments on synthetic traces and real data sets that validate the robustness and accuracy of our algorithms.

Online Scheduling for Shuffle Grouping in Distributed Stream Processing Systems

Shuffle grouping is a technique used by stream processing frameworks to share input load among parallel instances of stateless operators. With shuffle grouping each tuple of a stream can be assigned to any available operator instance, independently from any previous assignment. A common approach to implement shuffle grouping is to adopt a Round-Robin policy, a simple solution that fares well as long as the tuple execution time is almost the same for all the tuples. However, such an assumption rarely holds in real cases where execution time strongly depends on tuple content. As a consequence, parallel stateless operators within stream processing applications may experience unpredictable unbalance that, in the end, causes undesirable increase in tuple completion times. In this paper we propose Online Shuffle Grouping (OSG), a novel approach to shuffle grouping aimed at reducing the overall tuple completion time. OSG estimates the execution time of each tuple, enabling a proactive and online scheduling of input load to the target operator instances. Sketches are used to efficiently store the otherwise large amount of information required to schedule incoming load. We provide a probabilistic analysis and illustrate, through both simulations and a running prototype, its impact on stream processing applications.

Trust Evaluation of a System for an Activity

When users need to perform a digital activity, they evaluate available systems according to their functionality, ease of use, QoS, and/or economical aspects. Recently, trust has become another key factor for such evaluation. Two main issues arise in the trust management research community. First, how to define the trust in an entity, knowing that this can be a person, a digital or a physical resource. Second, how to evaluate such value of trust in a system as a whole for a particular activity. Defining and evaluating trust in systems is an open problem because there is no consensus on the used approach. In this work we propose an approach applicable to any kind of system. The distinctive feature of our proposal is that, besides taking into account the trust in the different entities the user depends on to perform an activity, it takes into consideration the architecture of the system to determine its trust level. Our goal is to enable users to have a personal comparison between different systems for the same application needs and to choose the one satisfying their expectations. This paper introduces our approach, which is based on probability theory, and presents ongoing results.

An Information Divergence Estimation over Data Streams

In this paper, we consider the setting of large scale distributed systems, in which each node needs to quickly process a huge amount of data received in the form of a stream that may have been tampered with by an adversary. In this situation, a fundamental problem is how to detect and quantify the amount of work performed by the adversary. To address this issue, we have proposed in a prior work, AnKLe, a one pass algorithm for estimating the Kullback-Leibler divergence of an observed stream compared to the expected one. Experimental evaluations have shown that the estimation provided by AnKLe is accurate for different adversarial settings for which the quality of other methods dramatically decreases. In the present paper, considering n as the number of distinct data items in a stream, we show that AnKLe is an (ε, δ)-approximation algorithm with a space complexity Õ(1/ε + 1/ε2) bits in “most” cases, and Õ(1/ε + n-ε-1/ε2 ) otherwise. To the best of our knowledge, an approximation algorithm for estimating the Kullback-Leibler divergence has never been analyzed before.

SocioPath: Bridging the Gap between Digital and Social Worlds

Everyday, people use more and more digital resources (data, application systems, Internet, etc.) for all aspects of their life, like financial management, private exchanges, collaborative work, etc. This leads to non-negligible dependences on the digital distributed resources that reveal strong reliance at the social level. Users are often not aware of their real autonomy regarding the management of their digital resources. People underestimate social dependences generated by the system they use and the resulting potential risks. We argue that it is necessary to be aware of some key aspects of system’s architectures to be able to know dependences. This work proposes SocioPath, a generic meta-model to derive dependences generated by system’s architectures. It focuses on relations, like access, control, ownership among different entities of the system (digital resources, hardware, persons, etc.). Enriched with deduction rules and definitions, SocioPath reveals the dependence of a person on each entity in the system. This meta-model can be useful to evaluate a system, as a modeling tool that bridges the gap between the digital and the social worlds.

SOLIST or How to Look for a Needle in a Haystack? A Lightweight Multi-overlay Structure for Wireless Sensor Networks

In this paper, we consider sensor database systems. Sensors are attached to objects and queries on the objects are operated at the sensor network level. Although queries to such a system might be extremely complex, ensuring efficiently basic functionalities such as broadcast or anycast without any central element is not trivial. In this paper, we provide a suite of *-cast (anycast, k-cast, broadcast) functionalities in a fully decentralized manner. More specifically, we present the design and evaluation of SOLIST, a multi-layer structure for sensors, largely inspired from structured peer-to-peer systems providing such functionalities. The main goal of SOLIST is to limit the overall energy consumption. A type is associated to each sensor, and the *-cast functionalities are implemented at a type granularity regardless of the number of types and their distribution within the network. A typical use of such a system is sensor-based stock management. We evaluate SOLIST through simulations and show that SOLIST achieves a reasonable trade-off between performance and energy consumption.