Nicolo Rivetti

REMI: A framework of reusable elements for mining heterogeneous data with missing information: A Tale of Congestion in Two Smart Cities

Applications targeting smart cities tackle common challenges, however solutions are seldom portable from one city to another due to the heterogeneity of smart city ecosystems. A major obstacle involves the differences in the levels of available information. In this work, we present REMI, which is a mining framework that handles varying degrees of information availability by providing a meta-solution to missing data. The framework core concept is the REMI layered stack architecture, offering two complementary approaches to dealing with missing information, namely data enrichment (DARE) and graceful degradation (GRADE). DARE aims at inference of missing information levels, while GRADE attempts to mine the patterns using only the existing data.We show that REMI provides multiple ways for re-usability, while being fault tolerant and enabling incremental development. One may apply the architecture to different problem instantiations within the same domain, or deploy it across various domains. Furthermore, we introduce the other three components of the REMI framework backing the layered stack. To support decision making in this framework, we show a mapping of REMI into an optimization problem (OTP) that balances the trade-off between three costs: inaccuracies in inference of missing data (DARE), errors when using less information (GRADE), and gathering of additional data. Further, we provide an experimental evaluation of REMI using real-world transportation data coming from two European smart cities, namely Dublin and Warsaw.

Venilia, On-line Learning and Prediction of Vessel Destination

The ACM DEBS 2018 Grand Challenge focuses on (soft) real-time prediction of both the destination port and the time of arrival of vessels, monitored through the Automated Identification System (AIS). Venilia prediction mechanism is based on a variety of machine learning techniques, including Markov predictive models. To improve the accuracy of a model, trained off-line on historical data, Venilia supports also on-line continuous training using an incoming event stream. The software architecture enables a low latency, highly parallelized, and load balanced prediction pipeline. Aiming at a portable and reusable solution, Venilia is implemented on top of the Akka Actor framework. Finally, Venilia is also equipped with a visualization tool for data exploration.

Probabilistic Management of Late Arrival of Events

In a networked world, events are transmitted from multiple distributed sources into CEP systems, where events are related to one another along multiple dimensions, e.g., temporal and spatial, to create complex events. The big data era brought with it an increase in the scale and frequency of event reporting. Internet of Things adds another layer of complexity with multiple, continuously changing event sources, not all of which are perfectly reliable, often suffering from late arrivals. In this work we propose a probabilistic model to deal with the problem of reduced reliability of event arrival time. We use statistical theories to fit the distributions of inter-generation at the source and network delays per event type. Equipped with these distributions we propose a predictive method for determining whether an event belonging to a window has yet to arrive. Given some user-defined tolerance levels (on quality and timeliness), we propose an algorithm for dynamically determining the amount of time a complex event time-window should remain open. Using a thorough empirical analysis, we compare the proposed algorithm against state-of-the-art mechanisms for delayed arrival of events and show the superiority of our proposed method.

FlinkMan: Anomaly Detection in Manufacturing Equipment with Apache Flink: Grand Challenge

We present a (soft) real-time event-based anomaly detection application for manufacturing equipment, built on top of the general purpose stream processing framework Apache Flink. The anomaly detection involves multiple CPUs and/or memory intensive tasks, such as clustering on large time-based window and parsing input data in RDF-format. The main goal is to reduce end-to-end latencies, while handling high input throughput and still provide exact results. Given a truly distributed setting, this challenge also entails careful task and/or data parallelization and balancing. We propose FlinkMan, a system that offers a generic and efficient solution, which maximizes the usage of available cores and balances the load among them. We illustrates the accuracy and efficiency of FlinkMan, over a 3-step pipelined data stream analysis, that includes clustering, modeling and querying.

Data Streaming and its Application to Stream Processing: Tutorial

In this tutorial paper we present the results of recent research findings in the area of data streaming applied to stream processing systems. In particular, we introduce the data streaming model, detailing the main algorithmic results in this research field. We then move to detail how such algorithms can be applied to modern distributed stream processing systems to improve their efficiency. Finally we outline several open research directions in this field.

REMI, Reusable Elements for Multi-Level Information Availability: Demo

Applications targeting Smart Cities tackle common challenges, however solutions are seldom portable from one city to another due to the heterogeneity of city ecosystems. A major obstacle involves the differences in the levels of available information. In this demonstration we present REMI, a reusable elements framework to handle varying degrees of information availability by design from two complementary angles, namely graceful degradation (GRADE) and data enrichment (DARE). In a nutshell, we develop reusable machine learning black boxes for mining and aggregating streaming data, either to infer missing data from available data, or to adapt expected accuracy based on data availability. We illustrate the proposed approach using tram data from the city of Warsaw.