Stefania Tosi

Adaptive, scalable and reliable monitoring of big data on clouds

Real-time monitoring of cloud resources is crucial for a variety of tasks such as performance analysis, workload management, capacity planning and fault detection. Applications producing big data make the monitoring task very difficult at high sampling frequencies because of high computational and communication overheads in collecting, storing, and managing information. We present an adaptive algorithm for monitoring big data applications that adapts the intervals of sampling and frequency of updates to data characteristics and administrator needs. Adaptivity allows us to limit computational and communication costs and to guarantee high reliability in capturing relevant load changes. Experimental evaluations performed on a large testbed show the ability of the proposed adaptive algorithm to reduce resource utilization and communication overhead of big data monitoring without penalizing the quality of data, and demonstrate our improvements to the state of the art. Real time monitoring of cloud resources is crucial for system management.We propose an adaptive algorithm for scalable and reliable cloud monitoring.Our algorithm dynamically balances amount and quality of monitored time series.We reduce monitoring costs significantly without penalizing data quality.

A Software Architecture for the Analysis of Large Sets of Data Streams in Cloud Infrastructures

System management algorithms in private and public cloud infrastructures have to work with literally thousands of data streams generated from resource, application and event monitors. This cloud context opens two novel issues that we address in this paper: how to design a software architecture that is able to gather and analyze all information within real-time constraints, how it is possible to reduce the analysis of the huge collected data set to the investigation of a reduced set of relevant information. The application of the proposed architecture is based on the most advanced software components, and is oriented to the classification of the statistical behavior of servers and to the analysis of significant state changes. These results guide model-driven management systems to investigate only relevant servers and to apply suitable decision models considering the deterministic or non-deterministic nature of server behaviors.

An adaptive model for online detection of relevant state changes in Internet-based systems

Modern Internet-based systems typically involve a large number of servers and applications and require real-time management strategies for cloning and migrating virtual machines, as well as re-distributing or re-mapping the underlying hardware. At the basis of most real-time management strategies there is the need to continuously evaluate system state behavior and to detect when a relevant state change is occurring. Modern Internet-based systems open new and interesting scenarios in the field of the research on the online state change detection models. In this paper, we propose an adaptive state change detection model that we demonstrate is suitable to analyze continuous streams of data coming from Internet-based systems characterized by high variability and non stationarity of the monitored resource measures that result in not-acceptable false alarm rates. Our model solves the limits of the traditional solutions while retaining their computational efficiency. The solution we present combines two key elements: an on-line wavelet model to denoise data streams and an adaptive detection rule. Experiments carried out using empirical and synthetic data sets confirm that the proposed method is able to signal all relevant state changes limiting the incorrect detections and to provide robust results even in non-stationary and highly variable contexts.

Real-time adaptive algorithm for resource monitoring

In large scale systems, real-time monitoring of hardware and software resources is a crucial means for any management purpose. In architectures consisting of thousands of servers and hundreds of thousands of component resources, the amount of data monitored at high sampling frequencies represents an overhead on system performance and communication, while reducing sampling may cause quality degradation. We present a real-time adaptive algorithm for scalable data monitoring that is able to adapt the frequency of sampling and data updating for a twofold goal: to minimize computational and communication costs, to guarantee that reduced samples do not affect the accuracy of information about resources. Experiments carried out on heterogeneous data traces referring to synthetic and real environments confirm that the proposed adaptive approach reduces utilization and communication overhead without penalizing the quality of data with respect to existing monitoring algorithms.

Selecting Optimum Cloud Availability Zones by Learning User Satisfaction Levels

Cloud service providers enable enterprises with the ability to place their business applications into availability zones across multiple locations worldwide. While this capability helps achieve higher availability with smaller failure rates, business applications deployed across these independent zones may experience different quality of service (QoS) due to heterogeneous physical infrastructures. Since the perceived QoS against specific requirements are not usually advertised by cloud providers, selecting an availability zone that would best satisfy the user requirements is a challenge. In this paper, we introduce a predictive approach to identify the cloud availability zone that maximizes satisfaction of an incoming request against a set of requirements. The prediction models are built from historical usage data for each availability zone and are updated as the nature of the zones and requests change. Simulation results show that our method successfully predicts the unpublished zone behavior from historical data and identifies the availability zone that maximizes user satisfaction against specific requirements.

Monitoring Large Cloud-Based Systems

Large scale cloud-based services are built upon a multitude of hardware and software resources, disseminated in one or multiple data centers. Controlling and managing these resources requires the integration of several pieces of software that may yield a representative view of the data center status. Todays both closed and open-source monitoring solutions fail in different ways, including the lack of scalability, scarce representativity global state conditions, inability in guaranteeing persistence in service delivery, and the impossibility of monitoring multi-tenant applications. In this paper, we present a novel monitoring architecture that addresses the aforementioned issues. It integrates a hierarchical scheme to monitor the resources in a cluster with a distributed hash table (DHT) to broadcast system state information among different monitors. This architecture strives to obtain high scalability, effectiveness and resilience, as well as the possibility of monitoring services spanning across different clusters or even different data centers of the cloud provider. We evaluate the scalability of the proposed architecture through a bottleneck analysis achieved by experimental results.

Real-time models supporting resource management decisions in highly variable systems

Data centers providing modern interactive applications are enriched by autonomous management decision systems that are able to clone and migrate virtual machines, to re-distribute resources or to re-map services in real-time. At the basis of all these decisions, there is the need of a continuous evaluation of the state of system resources and of detecting when some relevant changes are occurring. Unfortunately, the load of interactive applications reaching the system is intrinsically heterogeneous with consequent highly variable effects on the resource behavior emerging from system monitors. Hence, existing algorithms for online detection of state changes are affected by low precision and scarce robustness when they are applied to modern contexts. We propose a novel model for online detection of relevant state changes that combines a filtered representation of the raw measures with adaptive detection rules. Experiments carried out on real and emulated data sets confirm that the proposed model is able to timely signal all relevant state changes, to limit false detections and, even more important, its results are robust in highly variable contexts.

Self-Adaptive Techniques for the Load Trend Evaluation of Internal System Resources

Modern distributed systems that have to avoid performance degradation and system overload require several runtime management decisions for load balancing and load sharing, overload and admission control, job dispatching and request redirection. As the external workload and the internal resource behavior of the modern system is highly complex and variable, self-adaptive techniques require a stable vision of the system behavior. In this paper we propose a trend model that guarantees a robust interpretation for load-aware decision algorithms. Various experimental results in a Web cluster demonstrate that the proposed models and algorithms guarantee better stability of the load and a reduction of the response time experienced by the users.

A Predictive Method for Identifying Optimum Cloud Availability Zones

Cloud service providers enable enterprises with the ability to place their business applications into availability zones across multiple locations worldwide. While this capability helps achieve higher availability with smaller failure rates, business applications deployed across these independent zones may experience different Quality of Service (QoS) due to heterogeneous physical infrastructures. Since the perceived QoS against specific requirements are not usually advertised by cloud providers, selecting an availability zone that would best satisfy the user requirements is a challenge. In this paper, we introduce a predictive approach to identify the cloud availability zone that maximizes satisfaction of an incoming request against a set of requirements. The predictive models are built from historical usage data for each availability zone and are updated as the nature of the zones and requests change. Simulation results show that our method successfully predicts the unpublished zone behavior from historical data and identifies the availability zone that maximizes user satisfaction against specific requirements.

Data clustering based on correlation analysis applied to highly variable domains

Clustering of traffic data based on correlation analysis is an important element of several network management objectives including traffic shaping and quality of service control. Existing correlation-based clustering algorithms are affected by poor results when applied to highly variable time series characterizing most network traffic data. This paper proposes a new similarity measure for computing clusters of highly variable data on the basis of their correlation. Experimental evaluations on several synthetic and real datasets show the accuracy and robustness of the proposed solution that improves existing clustering methods based on statistical correlations.