Augusto Born de Oliveira

DataMill: rigorous performance evaluation made easy

Empirical systems research is facing a dilemma. Minor aspects of an experimental setup can have a significant impact on its associated performance measurements and potentially invalidate conclusions drawn from them. Examples of such influences, often called hidden factors, include binary link order, process environment size, compiler generated randomized symbol names, or group scheduler assignments. The growth in complexity and size of modern systems will further aggravate this dilemma, especially with the given time pressure of producing results. So how can one trust any reported empirical analysis of a new idea or concept in computer science? This paper introduces DataMill, a community-based easy-to-use services-oriented open benchmarking infrastructure for performance evaluation. DataMill facilitates producing robust, reliable, and reproducible results. The infrastructure incorporates the latest results on hidden factors and automates the variation of these factors. Multiple research groups already participate in DataMill. DataMill is also of interest for research on performance evaluation. The infrastructure supports quantifying the effect of hidden factors, disseminating the research results beyond mere reporting. It provides a platform for investigating interactions and composition of hidden factors.

Why you should care about quantile regression

Research has shown that correctly conducting and analysing computer performance experiments is difficult. This paper investigates what is necessary to conduct successful computer performance evaluation by attempting to repeat a prior experiment: the comparison between two Linux schedulers. In our efforts, we found that exploring an experimental space through a series of incremental experiments can be inconclusive, and there may be no indication of how much experimentation will be enough. Analysis of variance (ANOVA), a traditional analysis method, is able to partly solve the problems with the previous approach, but we demonstrate that ANOVA can be insufficient for proper analysis due to the requirements it imposes on the data. Finally, we demonstrate the successful application of quantile regression, a recent development in statistics, to computer performance experiments. Quantile regression can provide more insight into the experiment than ANOVA, with the additional benefit of being applicable to data from any distribution. This property makes it especially useful in our field, since non-normally distributed data is common in computer experiments.

Optimization-Based Dynamic Reconfiguration of Real-Time Schedulers With Support for Stochastic Processor Consumption

The complexity of real-time systems has substantially increased in the past few years regarding both hardware and software aspects. The use of modern sensors, able to capture image and audio data, demands predictable multimedia-like data processing. Moreover, applications like autonomous robots, surveillance, or modern multimedia players may well be characterized by several operation modes, each one associated with light conditions, vision angle, change in user requirements, etc. In this paper, we describe suitable scheduling mechanisms that address these aspects. Application modes are characterized by their required processing bandwidth and benefit values. By using bandwidth reservation schedulers, dynamic reconfiguring scheduling parameters is seen as an optimization problem whose goal is to maximize the overall system benefit subject to schedulability constraints. Two different models for the problem are defined, Discrete and Continuous. The former gives rise to an NP-Hard problem for which efficient approximate solutions are derived. An optimal and polynomial solution to the Continuous model is derived. Both models are then extended to incorporate task execution times described as probability distributions. Making use of this stochastic modeling one is able to dynamically reconfigure the scheduler subject to probabilistic schedulability guarantees. The derived solutions are evaluated by extensive simulation, which indicates the good performance of the proposed reconfiguration mechanisms.

D-RES: Correct transitive distributed service sharing

With the growth of complexity in the embedded domain, the use of distributed systems to support multiple realtime applications has become commonplace. These applications may share processor and network resources, and real-time scheduling policies can guarantee that these applications do not interfere with each others ability to meet their temporal constraints. We believe that these applications should also be able to transparently share services and chains of services, without the coupling that such sharing typically implies. To solve this problem, we propose D-RES, a resource management system that guarantees temporal isolation between service-sharing applications in a distributed system. D-RES transparently tracks which application uses which service, billing the correct application even in case of nested service calls. We implemented D-RES, and demonstrate its ability to isolate service-sharing applications even in case of overload.

Configurable Medium Access Control for Wireless Sensor Networks

This article presents C-MAC, a Configurable Protocol for medium access control in wireless sensor networks. C-MAC works as a framework of medium access control strategies, with a transparent configuration system. The protocol aggregates different services, each implemented in several different strategies. Applications may configure different communication parameters in compile-time and run-time. C-MAC’s metaprogrammed implementation yields smaller footprint and higher performance than equivalent protocols for wireless sensor networks.

mTags: augmenting microkernel messages with lightweight metadata

In this work we propose mTags, an efficient mechanism that augments microkernel interprocess messages with lightweight metadata to enable the development of new, system-wide functionality without requiring modification of the application source code. As such it is well suited for systems with a large legacy code base or third-party applications like phone and tablet applications. We explored mTags in a variety of different contexts in local and distributed system scenarios. For example, we detail use cases in areas including messaging-induced deadlocks and mode propagation. To demonstrate that mTags is technically feasible and practical, we implemented it in a commercial microkernel and executed multiple sets of standard benchmarks on two different computing architectures. The results clearly demonstrate that mTags has only negligible overhead and strong potential for many applications.

Perphecy: Performance Regression Test Selection Made Simple but Effective

Developers of performance sensitive production software are in a dilemma: performance regression tests are too costly to run at each commit, but skipping the tests delays and complicates performance regression detection. Ideally, developers would have a system that predicts whether a given commit is likely to impact performance and suggests which tests to run to detect a potential performance regression. Prior approaches towards this problem require static or dynamic analyses that limit their generality and applicability. This paper presents an approach that is simple and general, and that works surprisingly well for real applications.

DataMill: a distributed heterogeneous infrastructure forrobust experimentation

Empirical systems research is facing a dilemma. Minor aspects of an experimental setup can have a significant impact on its associated performance measurements and potentially invalidate conclusions drawn from them. Examples of such influences, often called hidden factors, include binary link order, process environment size, compiler generated randomized symbol names, or group scheduler assignments. The growth in complexity and size of modern systems will further aggravate this dilemma, especially with the given time pressure of producing results. How can one trust any reported empirical analysis of a new idea or concept in computer science? DataMill is a community‐based services‐oriented open benchmarking infrastructure for rigorous performance evaluation. DataMill facilitates producing robust, reliable, and reproducible results. The infrastructure incorporates the latest results on hidden factors and automates the variation of these factors. DataMill is also of interest for research on performance evaluation. The infrastructure supports quantifying the effect of hidden factors, disseminating the research results beyond mere reporting. It provides a platform for investigating interactions and composition of hidden factors. This paper discusses experience earned through creating and using an open benchmarking infrastructure. Multiple research groups participate and have used DataMill. Furthermore, DataMill has been used for a performance competition at the International Conference on Runtime Verification (RV) 2014 and is currently hosting the RV 2015 competition. This paper includes a summary of our experience hosting the first RV competition. Copyright

Intersert: Assertions on Distributed Process Interaction Sessions

Program assertions typically operate on available program state such as global and local variables. To support sophisticated assert statements such as invariants on control flow or inter-process communication patterns, developers must design and maintain supporting infrastructure. It is non-obvious how to realize this infrastructure: how to maintain the data, how to access it, how to use it in assertions, how to keep the overhead low enough for embedded systems, and how to manage assertions across a distributed system.This work demonstrates the utility of assertions on interaction history among distributed system components and solves the challenges of efficiently maintaining interaction data while providing an expressive interface for assertions. Our toolchain enables developers to program assertions on interaction history written in regular expressions that incorporate inter-process and inter-thread behavior amongst multiple components in a distributed system. We demonstrate that the interaction tracking and property verification systems incur negligible overhead, measured with several benchmarks. This work discusses our toolchain with a real-world safety-critical embedded system.

Integrating Wireless Sensor Networks and the Grid through POP-C++

The topic of interaction between Wireless Sensor Networks (WSNs) and other computation systems has received relatively low scientific attention, and the interface between the data source and the applications that use that data remains a problem for the application programmer. This work extends POP-C++, a programming language and runtime support system for Grid programming, to enable Grid applications to seamlessly and concurrently use WSNs’ sensing and processing capabilities.