Peter Libič

Utilizing Performance Unit Tests To Increase Performance Awareness

Many decisions taken during software development impact the resulting application performance. The key decisions whose potential impact is large are usually carefully weighed. In contrast, the same care is not used for many decisions whose individual impact is likely to be small -- simply because the costs would outweigh the benefits. Developer opinion is the common deciding factor for these cases, and our goal is to provide the developer with information that would help form such opinion, thus preventing performance loss due to the accumulated effect of many poor decisions. Our method turns performance unit tests into recipes for generating performance documentation. When the developer selects an interface and workload of interest, relevant performance documentation is generated interactively. This increases performance awareness -- with performance information available alongside standard interface documentation, developers should find it easier to take informed decisions even in situations where expensive performance evaluation is not practical. We demonstrate the method on multiple examples, which show how equipping code with performance unit tests works.

DOs and DON'Ts of Conducting Performance Measurements in Java

The tutorial aims at practitioners - researchers or developers - who need to execute small scale performance experiments in Java. The goal is to provide the attendees with a compact overview of some of the issues that can hinder the experiment or mislead the evaluation, and discuss the methods and tools that can help avoid such issues. The tutorial will examine multiple elements of the software execution stack that impact performance, including common virtual machine mechanisms (just-in-time compilation and garbage collection together with associated runtime adaptation), some operating system features (timers) and hardware (memory) - although the focus will be on Java, some of the take away points should apply even in a more general performance experiment context.

On the limits of modeling generational garbage collector performance

Garbage collection is an element of many contemporary software platforms whose performance is determined by complex interactions and is therefore difficult to quantify and model. We investigate the difference between the behavior of a real garbage collector implementation and a simplified model on a selection of workloads, focusing on the accuracy achievable with particular input information (sizes, references, lifetimes). Our work highlights the limits of performance modeling of garbage collection and points out issues of existing evaluation tools that may lead to incorrect experimental conclusions.

On the accuracy of cache sharing models

Memory caches significantly improve the performance of workloads that have temporal and spatial locality by providing faster access to data. Current processor designs have multiple cores sharing a cache. To accurately model a workload performance and to improve system throughput by intelligently scheduling workloads on cores, we need to understand how sharing caches between workloads affects their data accesses. Past research has developed analytical models that estimate the cache behavior for combined workloads given the stack distance profiles describing these workloads. We extend this research by presenting an analytical model with contributions to accuracy and composability - our model makes fewer simplifying assumptions than earlier models, and its output is in the same format as its input, which is an important property for hierarchical composition during software performance modeling. To compare the accuracy of our analytical model with earlier models, we attempted to reproduce the reported accuracy of those models. This proved to be difficult. We provide additional insight into the major factors that influence analytical model accuracy.

Issues in performance modeling of applications with garbage collection

Even though garbage collectors are incorporated in many service oriented systems, service performance models typically treat garbage collector overhead as a constant background factor. We use benchmark experiments to show that this treatment can make the service performance models miss performance effects of significant scale, and provides an initial inquiry into the issues related to including a generic garbage collector overhead model as a part of the service performance models.

Estimating the Impact of Code Additions on Garbage Collection Overhead

In managed memory environments, code changes influence performance both through time spent executing the code and time spent collecting garbage generated by the code. This complicates decision making when considering performance impact of code changes—while the impact on execution time is usually easy to assess in isolation, the impact on garbage collection time depends on the memory allocation behavior of the code surrounding the changes. In our paper, we describe a method to estimate the impact of code changes with additional allocations on garbage collection time, which can be applied, e.g., when assessing the overall performance impact of alternative changes. The method is demonstrated on experiments with the HotSpot virtual machine.

Timing penalties associated with cache sharing

Although important from software performance perspective, the behavior of memory caches is not captured by the common approaches to modeling of software performance, where the software performance models tend to treat operation durations as constants despite the fact that the operations compete for memory caches. Incorporating memory cache models into software performance models is hindered by the fact that existing cache models do not provide information about timings and penalties, but only about hits and misses. The paper outlines the relationship of cache events and cache timings on a real computer architecture, indicating that the existing practice of modeling cache miss penalties as constants is not sufficient to model software performance faithfully.

Analysis of Overhead in Dynamic Java Performance Monitoring

In production environments, runtime performance monitoring is often limited to logging of high level events. More detailed measurements, such as method level tracing, tend to be avoided because their overhead can disrupt execution. This limits the information available to developers when solving performance issues at code level. One approach that reduces the measurement disruptions is dynamic performance monitoring, where the measurement instrumentation is inserted and removed as needed. Such selective monitoring naturally reduces the aggregate overhead, but also introduces transient overhead artefacts related to insertion and removal of instrumentation. We experimentally analyze this overhead in Java, focusing in particular on the measurement accuracy, the character of the transient overhead, and the longevity of the overhead artefacts. Among other results, we show that dynamic monitoring requires time from seconds to minutes to deliver stable measurements, that the instrumentation can both slow down and speed up the execution, and that the overhead artefacts can persist beyond the monitoring period.