Jiang Li

Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis

Detection of execution anomalies is very important for the maintenance, development, and performance refinement of large scale distributed systems. Execution anomalies include both work flow errors and low performance problems. People often use system logs produced by distributed systems for troubleshooting and problem diagnosis. However, manually inspecting system logs to detect anomalies is unfeasible due to the increasing scale and complexity of distributed systems. Therefore, there is a great demand for automatic anomalies detection techniques based on log analysis. In this paper, we propose an unstructured log analysis technique for anomalies detection. In the technique, we propose a novel algorithm to convert free form text messages in log files to log keys without heavily relying on application specific knowledge. The log keys correspond to the log-print statements in the source code which can provide cues of system execution behavior. After converting log messages to log keys, we learn a Finite State Automaton (FSA) from training log sequences to present the normal work flow for each system component. At the same time, a performance measurement model is learned to characterize the normal execution performance based on the log mes-sages’ timing information. With these learned models, we can automatically detect anomalies in newly input log files. Experiments on Hadoop and SILK show that the technique can effectively detect running anomalies.

An Epipolar Geometry-Based Fast Disparity Estimation Algorithm for Multiview Image and Video Coding

Effectively coding multiview visual content is an indispensable research topic because multiview image and video that provide greatly enhanced viewing experiences often contain huge amounts of data. Generally, conventional hybrid predictive-coding methodologies are adopted to address the compression by exploiting the temporal and interviewpoint redundancy existing in a multiview image or video sequences. However, their key yet time-consuming component, motion estimation (ME), is usually not efficient in interviewpoint prediction or disparity estimation (DE), because interviewpoint disparity is completely different from temporal motion existing in the conventional video. Targeting a generic fast DE framework for interviewpoint prediction, we propose a novel DE technique in this paper to accelerate the disparity search by employing epipolar geometry. Theoretical analysis, optimal disparity vector distribution histograms, and experimental results show that the proposed epipolar geometry-based DE can greatly reduce search region and effectively track large and irregular disparity, which is typical in convergent multiview camera setups. Compared with the existing state-of-the-art fast ME approaches, our proposed DE can obtain a similar coding efficiency while achieving a significant speedup for interviewpoint prediction and coding. Moreover, a robustness study shows that the proposed DE algorithm is insensitive to the epipolar geometry estimation noise. Hence, its wide application for multiview image and video coding is promising

Mining dependency in distributed systems through unstructured logs analysis

Dependencies among system components are crucial to locating root errors in a distributed system. In this paper, we propose an approach to mine intercomponent dependencies from unstructured logs. The technique requires neither additional system instrumentation nor any application specific knowledge. In the approach, we first parse each log message into its log key and parameters. Then, we find dependent log key pairs belong to different components by leveraging co-occurrence analysis and parameter correspondence. After that, we use Bayesian decision theory to estimate the dependency direction of each dependent log key pair. We further apply time delay consistency to remove false positive detections. Case studies on Hadoop show that the technique successfully identifies the dependencies among the distributed system components.

Mining program workflow from interleaved traces

Successful software maintenance is becoming increasingly critical due to the increasing dependence of our society and economy on software systems. One key problem of software maintenance is the difficulty in understanding the evolving software systems. Program workflows can help system operators and administrators to understand system behaviors and verify system executions so as to greatly facilitate system maintenance. In this paper, we propose an algorithm to automatically discover program workflows from event traces that record system events during system execution. Different from existing workflow mining algorithms, our approach can construct concurrent workflows from traces of interleaved events. Our workflow mining approach is a three-step coarse-to-fine algorithm. At first, we mine temporal dependencies for each pair of events. Then, based on the mined pair-wise tem-poral dependencies, we construct a basic workflow model by a breadth-first path pruning algorithm. After that, we refine the workflow by verifying it with all training event traces. The re-finement algorithm tries to find out a workflow that can interpret all event traces with minimal state transitions and threads. The results of both simulation data and real program data show that our algorithm is highly effective.

Multiview Image Coding Based on Geometric Prediction

Many existing multiview image/video coding techniques remove inter-viewpoint redundancy by applying disparity compensation in a conventional video coding framework, e.g., H.264/MPEG-4 AVC. However, conventional methodology works ineffectively as it ignores the special characteristics of inter-viewpoint disparity. In this paper, we propose a geometric prediction methodology for accurate disparity vector (DV) prediction, such that we can largely reduce the disparity compensation cost. Based on the new DV predictor, we design a basic framework that can be implemented in most existing multiview image/video coding schemes. We also use state-of-the-art H.264/MPEG-4 AVC as an example to illustrate how the proposed framework can be integrated with conventional video coding algorithms. Our experiments show proposed scheme can effectively tracks disparity and greatly improves coding performance. Compared with H.264/MPEG-4 AVC codec, our scheme outperforms maximally 1.5 dB when encoding some typical multiview image sequences. We also carry out an experiment to evaluate the robustness of our algorithm. The results indicate our method is robust and can be used in practical applications.

DigiMetro - an application-level multicast system for multi-party video conferencing

The increasing demand for multi-party videoconferencing has aroused the research interest in the underlying multicast support. In this paper, we propose DigiMetro, an application-level multicast system tailored to small and impromptu videoconferencing. Breaking through the conventional wisdom to use shared overlay to handle multiple data sources, DigiMetro organizes the data delivery routes as source-specific trees, which are first constructed by a local greedy algorithm and then gradually improved by a global refinement procedure. Extensive simulation experiments demonstrate the efficiency of both algorithms. Moreover, DigiMetro is able to handle different video bit rates and provide different services over voice/video streams.

Distributed Density Estimation Using Non-parametric Statistics

Learning the underlying model from distributed data is often useful for many distributed systems. In this paper, we study the problem of learning a non-parametric model from distributed observations. We propose a gossip-based distributed kernel density estimation algorithm and analyze the convergence and consistency of the estimation process. Furthermore, we extend our algorithm to distributed systems under communication and storage constraints by introducing a fast and efficient data reduction algorithm. Experiments show that our algorithm can estimate underlying density distribution accurately and robustly with only small communication and storage overhead.

Color Image Coding by using Inter-Color Correlation

Inter-color correlation between the luminance component and chrominance components has been utilized for color image coding for years. However, the correlation has not been clearly analyzed. In this paper, we analyze the inter-color correlation and answer two questions related to color image coding: (1) what kind of inter-color correlation exists in color images after the discrete wavelet transform?; and, (2) how strong is it? This analysis helps us to find a most suitable inter-color context and eventually leads to a new embedded color image codec. By using the discovered inter-color context, significant performance improvement can be achieved when encoding chrominance components.

Lossless image compression with tree coding of magnitude levels

With the rapid development of digital technology in consumer electronics, the demand to preserve raw image data for further editing or repeated compression is increasing. Traditional lossless image coders usually consist of computationally intensive modeling and entropy coding phases, therefore might not be suitable to mobile devices or scenarios with a strict real-time requirement. This paper presents a new image coding algorithm based on a simple architecture that is easy to model and encode the residual samples. In the proposed algorithm, each residual sample is separated into three parts: (1) a sign value, (2) a magnitude value, and (3) a magnitude level. A tree structure is then used to organize the magnitude levels. By simply coding the tree and the other two parts without any complicated modeling and entropy coding, good performance can be achieved with very low computational cost in the binary-uncoded mode. Moreover, with the aid of context-based arithmetic coding, the magnitude values are further compressed in the arithmetic-coded mode. This gives close performance to JPEG-LS and JPEG2000.

Multicast of Real-Time Multi-View Video

As a recently emerging service, multi-view video provides a new viewing experience with high degree of freedom. However, due to the huge data amounts transferred, multi-view videos delivery remains a daunting challenge. In this paper, we propose a multi-view video-streaming system based on IP multicast. It can support a large number of users while still keeping a high degree of interactivity and low bandwidth consumption. Based on a careful user study, we have developed two schemes: one is for automatic delivery and the other for on-demand delivery. In automatic delivery, a server periodically multicasts special effect snapshots at a certain time interval. In on-demand delivery, the server delivers the snapshots based on distribution of users requests. We conducted extensive experiments and user-experience studies to evaluate the proposed systems performance, and found that the system could provide satisfying multi-view video service for users on a large scale