Yukihiro Watanabe

Online failure prediction in cloud datacenters by real-time message pattern learning

Once failures occur in a cloud datacenter accommodating a large number of virtual resources, they tend to spread rapidly and widely, impacting on many cloud users (tenant owners). One of the best ways to prevent a failure from spreading in the system is identifying signs of the failure before its occurrence and deal with it proactively before it causes serious problems. Although several approaches have been proposed to predict failures by analyzing past system message logs and identifying the relationship between the messages and the failures, it is still difficult to automatically predict the failure for several reasons such as various types of log message formats or time gaps between message pattern learning and application of the identified patterns in real systems. Based on this understanding, we propose a new failure prediction method in this paper which learns message patterns as the signs of failure automatically by classifying messages by their similarity without depending on their format and re-Iearning of message patterns in frequently-changed configurations. We implemented our failure prediction method and evaluated it by using system log data recorded in an actual cloud datacenter. The experimental result shows that our approach predicted failures with 80% precision and covered 90% of failure occurrences.

Prediction of failure occurrence time based on system log message pattern learning

In order to avoid failures or diminish the impact of them, it is important to deal with them before its occurrence. Some existing approaches for online failure prediction are insufficient to handle the upcoming failures beforehand, because they cannot predict the failures early enough to execute workaround operations for failure. To solve this problem, we have developed a method to estimate the prediction period (the time period when a failure is expected to occur). Our method identifies the message patterns showing predictive signs of a certain failure through Bayesian learning from log messages and past failure reports. Using these patterns it predicts the occurrence of failures and their prediction period with sufficient interval. We conducted the evaluation of our approach with log data obtained from an actual system. The results shows that our method predicted the occurrence of failure with sufficient interval (60 minutes before the occurrence of failures) and sufficient accuracy (precision: over 0.7, recall: over 0.8).

Learning from Before and After Recovery to Detect Latent Misconfiguration

Preventing system failure in cloud has become more important as a result of the prevalence of cloud use for mission-critical applications. One of the major causes of system failure in clouds is misconfiguration, as shown in recent studies. Hence, it is essential first to detect misconfiguration before it causes outage or degradation of service. Although cloud provides us flexible and auto-configurable infrastructure for expeditious implementation of systems, this also provokes frequent changes and complexity of the implementation, and leads to difficulty in verifying its configuration. In this paper, we present a method to detect latent misconfigurations. Our method is designed on the basis of our misconfiguration categorizations which gives us the capability to choose detection tactics by misconfiguration pattern, so the administrator can diagnose with less knowledge of configuration details. By generalized preprocessing of configuration data in which configuration files are input as-is, our method does not limit its target to a specific type of component. This enables us to diagnose system-wide misconfiguration while system configuration is frequently changed. The results of our experiment show that misconfiguration of a single configuration parameter is detected with over 90% F-measure.

Failure Prediction for Cloud Datacenter by Hybrid Message Pattern Learning

In operations and management of large-scale cloud data enters, it is essential for administrators to handle failures occurring in their infrastructure before causing service-level violations. Some techniques for failure prediction have been studied because they can be used to start the troubleshooting process at the early stage of troubles and to prevent service-level violations from occurring. By its nature, however, failure prediction involves a certain amount of incorrect detection (false-positive). When applying failure prediction to the operation and management of cloud data centers, incorrect detection can result in the execution of unnecessary workaround tasks and additional costs. Existing methods for failure prediction using Bayesian inference to identify message patterns related to a certain failure are difficult to apply to relatively stable systems, because the accuracy of their predictions deteriorates in environments where failure rarely occurs. In order to solve this problem, we propose a novel method to improve the accuracy of failure prediction by suppressing incorrect detections using a hybrid score that integrates the probability of simultaneous occurrence between a message pattern and a failure and frequency of the message patterns for the failure. We implemented this method and evaluated the accuracy in a real commercial cloud data enter. The evaluation results revealed that it improved the accuracy of failure prediction by 31.9% compared with the existing method in terms of precision in the best case.