Rui Ren

LogMaster: Mining Event Correlations in Logs of Large-Scale Cluster Systems

This paper presents a set of innovative algorithms and a system, named Log Master, for mining correlations of events that have multiple attributions, i.e., node ID, application ID, event type, and event severity, in logs of large-scale cloud and HPC systems. Different from traditional transactional data, e.g., supermarket purchases, system logs have their unique characteristics, and hence we propose several innovative approaches to mining their correlations. We parse logs into an n-ary sequence where each event is identified by an informative nine-tuple. We propose a set of enhanced apriori-like algorithms for improving sequence mining efficiency, we propose an innovative abstraction-event correlation graphs (ECGs) to represent event correlations, and present an ECGs-based algorithm for fast predicting events. The experimental results on three logs of production cloud and HPC systems, varying from 433490 entries to 4747963 entries, show that our method can predict failures with a high precision and an acceptable recall rates.

Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks.

Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change of input distribution, thus results in poor generalization, and aggravates the internal covariate shift which slows down the training. To bound dot product and decrease the variance, we propose to use cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot product in neural networks, which we call cosine normalization. We compare cosine normalization with batch, weight and layer normalization in fully-connected neural networks, convolutional networks on the data sets of MNIST, 20NEWS GROUP, CIFAR-10/100, SVHN. Experiments show that cosine normalization achieves better performance than other normalization techniques.

Digging deeper into cluster system logs for failure prediction and root cause diagnosis

As the sizes of supercomputers and data centers grow towards exascale, failures become normal. System logs play a critical role in the increasingly complex tasks of automatic failure prediction and diagnosis. Many methods for failure prediction are based on analyzing event logs for large scale systems, but there is still neither a widely used one to predict failures based on both non-fatal and fatal events, nor a precise one that uses fine-grained information (such as failure type, node location, related application, and time of occurrence). A deeper and more precise log analysis technique is needed. We propose a three-step approach to draw out event dependencies and to identify failure-event generating processes. First, we cluster frequent event sequences into event groups based on common events. Then we infer causal dependencies between events in each event group. Finally, we extract failure rules based on the observation that events of the same event types, on the same nodes or from the same applications have similar operational behaviors. We use this rich information to improve failure prediction. Our approach semi-automates diagnosing the root causes of failure events, making it a valuable tool for system administrators.

Characterization and architectural implications of big data workloads

The previous major efforts on big data benchmark either propose a large amount of workloads (e.g. a recent comprehensive big data benchmark suite - BigDataBench [4]), which impose cognitive difficulty on workload characterization and serious benchmarking cost; or only select a few workloads according to so-called popularity[1], which lead to partial or biased observations.

BDTUne: Hierarchical correlation-based performance analysis and rule-based diagnosis for big data systems

Although big data systems are in widespread use and there have much research efforts for improving big data systems performance, efficiently analysing and diagnosing performance bottlenecks over these massively distributed systems remain a major challenge. In this paper, we propose a hierarchical correlation-based analysis and rule-based diagnostic approach for big data systems. The key approaches lie in identifying performance bottlenecks, classifying root causes, analyzing performance according to multi-level performance metrics, and setting diagnostic rules for performance tuning. Based on this approach, we have implemented BDTune — a lightweight, extensible and transparent tool that can provide valuable insights into performance of big data applications with a very low overhead. We also report our experience on how to use BDTune to conduct performance analysis and performance bottlenecks diagnosis, and demonstrate BDTune can help users find the performance bottlenecks and provide optimization recommendations.

Data motifs: a lens towards fully understanding big data and AI workloads

The complexity and diversity of big data and AI workloads make understanding them difficult and challenging. This paper proposes a new approachto modelling and characterizing big data and AI workloads. We consider each big data and AI workload as a pipeline of one or more classes of units of computation performed on different initial or intermediate data inputs. Each class of unit of computation captures the common requirements while being reasonably divorced from individual implementations, and hence we call it a data motif. For the first time, among a wide variety of big data and AI workloads, we identify eight data motifs that take up most of the run time of those workloads, including Matrix, Sampling, Logic, Transform, Set, Graph, Sort and Statistic. We implement the eight data motifs on different software stacks as the micro benchmarks of an open-source big data and AI benchmark suite --- BigDataBench 4.0 (publicly available from http://prof.ict.ac.cn/BigDataBench), and perform comprehensive characterization of those data motifs from perspective of data sizes, types, sources, and patterns as a lens towards fully understanding big data and AI workloads. We believe the eight data motifs are promising abstractions and tools for not only big data and AI benchmarking, but also domain-specific hardware and software co-design.

BigDataBench-S: An Open-Source Scientific Big Data Benchmark Suite

Data generated from modern scientific instrumentation have grown up to an unprecedented scale. Moreover, data formats and computational behaviors of scientific big data workloads are much more complex than those in Internet services. These two facts pose a serious challenge to scientific data management and analytics. Among many concerns, the first one is how to build a comprehensive and representative scientific big data benchmark suite. Previous benchmark efforts either focus on Internet areas (i.e. BigDataBench) or pay attention to a specific area (i.e. GeneBase). This paper presents our preliminary work on building a comprehensive scientific big data benchmark suite---BigDataBench-S. Also, we use BigDataBench-S to evaluate several general-purpose big data management systems specifically designed for Internet services applications. Our evaluation shows: these systems cannot achieve expected performance for many scientific workloads, especially for complex matrix computation, for the lack of appropriate mechanisms and policies on data storage, query optimization and support of distributed matrix computation.