Naser Ezzati-Jivan

A stateful approach to generate synthetic events from Kernel traces

We propose a generic synthetic event generator from kernel trace events. The proposed method makes use of patterns of system states and environment-independent semantic events rather than platform-specific raw events. This method can be applied to different kernel and user level trace formats. We use a state model to store intermediate states and events. This stateful method supports partial trace abstraction and enables users to seek and navigate through the trace events and to abstract out the desired part. Since it uses the current and previous values of the systemstates and hasmore knowledge of the underlying system execution, it can generate a wide range of synthetic events. One of the obvious applications of this method is the identification of system faults and problems that will appear later in this paper. We will discuss the architecture of the method, its implementation, and the performance results.

State History Tree: An Incremental Disk-Based Data Structure for Very Large Interval Data

In this paper, we propose the State History Tree, a disk-based data structure to manage large streaming interval data. The State History Tree provides an efficient way to store interval data on permanent storage with a logarithmic access time. The disk-based structure ensures that extremely large data sets can be accommodated. The State History Tree stores intervals in blocks on disk in a tree organization. Unlike other interval management data structures like R-Trees, our solution avoids re-balancing the nodes, speeding up the tree construction. The proposed method is implemented in Java, and evaluated using large data sets (up to one terabyte). Those data sets were obtained from the state intervals computed from system events traced with the LTTng kernel tracer. The evaluation results demonstrate the performance and efficiency of the method, as compared with other solutions to managing huge interval data sets.

A framework to compute statistics of system parameters from very large trace files

In this paper, we present a framework to compute, store and retrieve statistics of various system metrics from large traces in an efficient way. The proposed framework allows for rapid interactive queries about system metrics values for any given time interval. In the proposed framework, efficient data structures and algorithms are designed to achieve a reasonable query time while utilizing less disk space. A parameter termed granularity degree (GD) is defined to determine the threshold of how often it is required to store the precomputed statistics on disk. The solution supports the hierarchy of system resources and also different granularities of time ranges. We explain the architecture of the framework and show how it can be used to efficiently compute and extract the CPU usage and other system metrics. The importance of the framework and its different applications are shown and evaluated in this paper.

Efficient Model to Query and Visualize the System States Extracted from Trace Data

Any application, database server, telephony server, or operating system maintains different states for their internal elements and resources. When tracing is enabled on such systems, the corresponding events in the trace logs can be used to extract and model the different state values of the traced modules to analyze their runtime behavior. In this paper, a generic method and corresponding data structures are proposed to model and manage the system state values, allowing efficient storage and access. The proposed state organization mechanism generates state intervals from trace events and stores them in a tree-based state history database. The state history database can then be used to extract the state of any system resources (i.e. cpu, process, memory, file, etc.) at any timestamp. The extracted state values can be used to track system problems (e.g. performance degradation). The proposed system is usable in both the offline tracing mode, when there is a set of trace files, and online tracing mode, when there is a stream of trace events. The proposed system has been implemented and used to display and analyze interactively various information extracted from very large traces in the magnitude order of 1 TB.

Cube data model for multilevel statistics computation of live execution traces

Execution trace logs are used to analyze system run‐time behaviour and detect problems. Trace analysis tools usually read the input logs and gather either a detailed or brief summary of them to later process and inspect in the analysis steps. However, continuous and lengthy trace streams contained in the live tracing mode make it difficult to indefinitely record all events or even a detailed summary of the whole stream. This situation is further complicated when the system aims to compare different parts of the trace and provide a multilevel and multidimensional analysis. This paper presents an architecture with corresponding data structures and algorithms to process stream events, generate an adequate summary—detailed enough for recent data and succinct enough for old data—and organize them to enable an efficient multilevel and multidimensional analysis, similar to online analytical processing analyses in the database applications. The proposed solution arranges data in a compact manner using interval forms and enables the range queries for any arbitrary time durations. Because this feature makes it possible to compare of different system parameters in different time areas, it significantly influences the systems ability to provide a comprehensive trace analysis. Although the Linux operating system trace logs are used to evaluate the solution, we propose a generic architecture that can be used to summarize various types of stream data. Copyright

Multilevel label placement for execution trace events

Automatic label assignment to graphical objects is an important problem in many applications such as cartography, online maps and graph drawings. In this paper, we present efficient algorithms for automatic label assignment to execution trace items (points or lines) in a trace visualization tool. The proposed label assignment algorithms aim to maximize the number of labeled items as well as increase the quality of assignments. The algorithms take into account both the topological and semantic relationships (e.g. level of details, repetitiveness, etc.) between the trace items in order to achieve assignments that are both quantitative and qualitative. The proposed method also supports assigning multiple labels to each trace item. The algorithms have been implemented and applied to different input traces. The experimental results show that considering the relationships between data items increases the labeling success rate and the quality.

AN EFFICIENT ANALYSIS APPROACH FOR MULTI-CORE SYSTEM TRACING DATA

Trace data usually contains information about the underlying system execution such as running processes, memory usage, disk and file accesses and other runtime information. However, this raw data is not what the system administrators are looking for. For instance, in a multicore system, the system administrator may need to know what is the utilization of each core or even what are the performance bottlenecks of the system. This analytical information is not available directly from the trace data and, in fact, is hidden behind the mountains of trace raw data. The trace data needs to be analyzed to extract the valuable information. Thus, efficient trace analysis tools and techniques need to be developed to handle large trace data and extract and provide useful and analytical information. In this paper, we propose a stateful trace analysis and abstrac tion approach that shares the computation and storage of the common information between parallel abstraction processes. This technique leads to relatively simple patterns compared to other pattern-based techniques. Furthermore, since it computes and stores the common state once, shared between related processes, it has a better computation and storage efficiency than stateless methods. The architectur e of the method, its applications, and evaluation are detaile d in this paper.

A declarative framework for stateful analysis of execution traces

With newer complex multi-core systems, it is important to understand an application’s runtime behavior to be able to debug its execution, detect possible problems and bottlenecks and finally identify potential root causes. Execution traces usually contain precise data about an application execution. Their analysis and abstraction at multiple levels can provide valuable information and insights about an application’s runtime behavior. However, with multiple abstraction levels, it becomes increasingly difficult to find the exact location of detected performance or security problems. Tracing tools provide various analysis views to help users to understand their application problems. However, these pre-defined views are often not sufficient to reveal all analysis aspects of the underlying application. A declarative approach that enables users to specify and build their own custom analysis and views based on their knowledge, requirements and problems can be more useful and effective. In this paper, we propose a generic declarative trace analysis framework to analyze, comprehend and visualize execution traces. This enhanced framework builds custom analyses based on a specified modeled state, extracted from a system execution trace and stored in a special purpose database. The proposed solution enables users to first define their different analysis models based on their application and requirements, then visualize these models in many alternate representations (Gantt chart, XY chart, etc.), and finally filter the data to get some highlights or detect some potential patterns. Several sample applications with different operating systems are shown, using trace events gathered from Linux and Windows, at the kernel and user-space levels.

Hardware-assisted software event tracing

Event tracing is a reliable and a low‐intrusiveness method to debug and optimize systems and processes. Low overhead is particularly important in embedded systems where resources and energy consumption is critical. The most advanced tracing infrastructures achieve a very low footprint on the traced software, bringing each tracepoint overhead to less than a microsecond. To reduce this still non‐negligible impact, the use of dedicated hardware resources is promising. In this paper, we propose complementary methods for tracing that rely on hardware modules to assist software tracing. We designed solutions to take advantage of CoreSight STM, CoreSight ETM, and Intel BTS, which are present on most newer ARM‐based systems‐on‐chip and Intel x86 processors. Our results show that the time overhead for tracing can be reduced by up to 10 times when assisted by hardware, as compared to software tracing with LTTng, a high‐performance tracer for Linux. We also propose a modification to the Perf tool to speed BTS execution tracing up to 65%.

Multi-scale navigation of large trace data: A survey

Dynamic analysis through execution traces is frequently used to analyze the runtime behavior of software systems. However, tracing long running executions generates voluminous data, which are complicated to analyze and manage. Extracting interesting performance or correctness characteristics out of large traces of data from several processes and threads is a challenging task. Trace abstraction and visualization are potential solutions to alleviate this challenge. Several efforts have been made over the years in many subfields of computer science for trace data collection, maintenance, analysis, and visualization. Many analyses start with an inspection of an overview of the trace, before digging deeper and studying more focused and detailed data. These techniques are common and well supported in geographical information systems, automatically adjusting the level of details depending on the scale. However, most trace visualization tools operate at a single level of representation, which are not adequate to support multilevel analysis. Sophisticated techniques and heuristics are needed to address this problem. Multi‐scale (multilevel) visualization with support for zoom and focus operations is an effective way to enable this kind of analysis. Considerable research and several surveys are proposed in the literature in the field of trace visualization. However, multi‐scale visualization has yet received little attention. In this paper, we provide a survey and methodological structure for categorizing tools and techniques aiming at multi‐scale abstraction and visualization of execution trace data and discuss the requirements and challenges faced to be able to meet evolving user demands.