Masataka Sonoda

Online failure prediction in cloud datacenters by real-time message pattern learning

Once failures occur in a cloud datacenter accommodating a large number of virtual resources, they tend to spread rapidly and widely, impacting on many cloud users (tenant owners). One of the best ways to prevent a failure from spreading in the system is identifying signs of the failure before its occurrence and deal with it proactively before it causes serious problems. Although several approaches have been proposed to predict failures by analyzing past system message logs and identifying the relationship between the messages and the failures, it is still difficult to automatically predict the failure for several reasons such as various types of log message formats or time gaps between message pattern learning and application of the identified patterns in real systems. Based on this understanding, we propose a new failure prediction method in this paper which learns message patterns as the signs of failure automatically by classifying messages by their similarity without depending on their format and re-Iearning of message patterns in frequently-changed configurations. We implemented our failure prediction method and evaluated it by using system log data recorded in an actual cloud datacenter. The experimental result shows that our approach predicted failures with 80% precision and covered 90% of failure occurrences.

Prediction of failure occurrence time based on system log message pattern learning

In order to avoid failures or diminish the impact of them, it is important to deal with them before its occurrence. Some existing approaches for online failure prediction are insufficient to handle the upcoming failures beforehand, because they cannot predict the failures early enough to execute workaround operations for failure. To solve this problem, we have developed a method to estimate the prediction period (the time period when a failure is expected to occur). Our method identifies the message patterns showing predictive signs of a certain failure through Bayesian learning from log messages and past failure reports. Using these patterns it predicts the occurrence of failures and their prediction period with sufficient interval. We conducted the evaluation of our approach with log data obtained from an actual system. The results shows that our method predicted the occurrence of failure with sufficient interval (60 minutes before the occurrence of failures) and sufficient accuracy (precision: over 0.7, recall: over 0.8).