Mathieu Desnoyers

User-Level Implementations of Read-Copy Update

Read-copy update (RCU) is a synchronization technique that often replaces reader-writer locking because RCUs read-side primitives are both wait-free and an order of magnitude faster than uncontended locking. Although RCU updates are relatively heavy weight, the importance of read-side performance is increasing as computing systems become more responsive to changes in their environments. RCU is heavily used in several kernel-level environments. Unfortunately, kernel-level implementations use facilities that are often unavailable to user applications. The few prior user-level RCU implementations either provided inefficient read-side primitives or restricted the application architecture. This paper fills this gap by describing efficient and flexible RCU implementations based on primitives commonly available to user-level applications. Finally, this paper compares these RCU implementations with each other and with standard locking, which enables choosing the best mechanism for a given workload. This work opens the door to widespread user-application use of RCU.

Lockless multi-core high-throughput buffering scheme for kernel tracing

Studying execution of concurrent real-time online systems, to identify far-reaching and hard to reproduce latency and performance problems, requires a mechanism able to cope with voluminous information extracted from execution traces. Furthermore, the workload must not be disturbed by tracing, thereby causing the problematic behavior to become unreproducible. In order to satisfy this low-disturbance constraint, we created the LTTng kernel tracer. It is designed to enable safe and race-free attachment of probes virtually anywhere in the operating system, including sites executed in non-maskable interrupt context. In addition to being reentrant with respect to all kernel execution contexts, LTTng offers good performance and scalability, mainly due to its use of per-CPU data structures, local atomic operations as main buffer synchronization primitive, and RCU (Read-Copy Update) mechanism to control tracing. Given that kernel infrastructure used by the tracer could lead to infinite recursion if traced, and typically requires non-atomic synchronization, this paper proposes an asynchronous mechanism to inform the kernel that a buffer is ready to read. This ensures that tracing sites do not require any kernel primitive, and therefore protects from infinite recursion. This paper presents the core of LTTngs buffering algorithms and measures its performance.

Experiences understanding performance in acommercial scale-out environment

Clusters of loosely connected machines are becoming an important model for commercial computing. The cost/performance ratio makes these scale-out solutions an attractive platform for a class of computational needs. The work we describe in this paper focuses on understanding performance when using a scale-out environment to run commercial workloads. We describe the novel scale-out environment we configured and the workload we ran on it. We explain the unique performance challenges faced in such an environment and the tools we applied and improved for this environment to address the challenges. We present data from the tools that proved useful in optimizing performance on our system. We discuss the lessons we learned applying and modifying existing tools to a commercial scale-out environment, and offer insights into making future performance tools effective in this environment.

Performance analysis of virtual machines through tracing

This paper presents extensions to the Linux trace toolkit (LTT) for efficiently tracing complete systems, from the user level to the virtual machine system level and the Xen hypervisor layer. An automated benchmark was created to measure the overhead imposed by tracing and Xen paravirtualization, comparing the performance of systems, with and without virtualization and tracing. The results show a negligible impact caused by tracing and a reasonable overhead caused by paravirtualization. For some I/O intensive applications, the overhead imposed by paravirtualization is less negligible.

Multi-core systems modeling for formal verification of parallel algorithms

Modeling parallel algorithms at the architecture level enables exploring side-effects of the weakly ordered nature of modern processors. Formal verification of such models with model-checking can ensure that algorithm guarantees will hold even in the presence of the most aggressive compiler and processor optimizations. This paper proposes a virtual architecture to model the effects of such optimizations. It first presents the OoOmem framework to model out-of-order memory accesses. It then presents the OoOisched framework to model the effects of out-of-order instruction scheduling. These two frameworks are explained and tested using weaklyordered memory interaction scenarios known to be affected by weak ordering. Then, modeling of user-level RCU (Read- Copy Update) synchronization algorithms is presented. It uses the virtual architecture proposed to verify that the RCU guarantees are indeed respected.

Tracing Time Operating System State Determination

In recent years, tracing operating system behavior by recording kernel events has proven to be a particularly ef- fective tool. However, when used to characterize the systems behavior through time, including its state, the list of state transitions that the kernel events represent is not sufficient to characterize the state for the entire data acquisition period. The initial operating system state, when tracing starts, is also required. The challenge lies in obtaining a complete snapshot of the initial state, while minimizing the impact on the system being traced. This impact may be in terms of CPU or disk I/O consumption, instrumentation memory, or burst activity at trace start time detrimental to the real-time response. Such impact is especially disturbing on small real-time limited resources embedded systems. In this paper, we will propose an efficient approach to extract such initial state information and discuss the software mod- ule we have developed to provide the aforementioned data to the LTTng tracing tool and its accompanying viewer, LTTV. This module not only improves LTTVs accuracy by providing the initial state of all processes in the system, but also pro- vides an inventory of relevant kernel objects at minimal cost, without increasing noticeably the interrupt latency.

Runtime latency detection and analysis

Detecting latency‐related problems in production environments is usually carried out at the application level with custom instrumentation. This is enough to detect high latencies in instrumented applications but does not provide all the information required to understand the source of the latency and is dependent on manually deployed instrumentation. The abnormal latencies usually start in the operating system kernel because of contention on physical resources or locks. Hence, finding the root cause of a latency may require a kernel trace. This trace can easily represent hundreds of thousands of events per second. In this paper, we propose and evaluate a methodology, efficient algorithms, and concurrent data structures to detect and analyze latency problems that occur at the kernel level. We introduce a new kernel‐based approach that enables developers and administrators to efficiently track latency problems in production and trigger actions when abnormal conditions are detected. The result of this study is a working scalable latency tracker and an efficient approach to perform stateful tracing in production. Copyright

Virtual Systems Tracing for Performance Analysis

This paper describes extensions to the Linux Trace Toolkit next generation (LTTng), to trace the Xen hypervisor for efficiently tracing complete systems. LTTng is a low impact, modular, extensible, operating system tracer, while Xen is an hypervisor based on paravirtualization. Typically, one or several instances of Linux will run on top of the Xen hypervisor. The extended LTTng is able to trace all Linux instances and the hypervisor layer to give a global view of the system. An automated benchmark was created to measure the overhead imposed by tracing and Xen paravirtualization. It compares the performance of systems, with and without virtualization and tracing. The results obtained with different tests, using this procedure, are presented and show a negligible impact caused by tracing and a reasonable overhead caused by paravirtualization. Nonetheless, for some I/O intensive applications, the overhead imposed by paravirtualization is less negligible.