Terence Kelly

Exploiting nonstationarity for performance prediction

Real production applications ranging from enterprise applications to large e-commerce sites share a crucial but seldom-noted characteristic: The relative frequencies of transaction types in their workloads are nonstationary, i.e., the transaction mix changes over time. Accurately predicting application-level performance in business-critical production applications is an increasingly important problem. However, transaction mix nonstationarity casts doubt on the practical usefulness of prediction methods that ignore this phenomenon. This paper demonstrates that transaction mix nonstationarity enables a new approach to predicting application-level performance as a function of transaction mix. We exploit nonstationarity to circumvent the need for invasive instrumentation and controlled benchmarking during model calibration; our approach relies solely on lightweight passive measurements that are routinely collected in todays production environments. We evaluate predictive accuracy on two real business-critical production applications. The accuracy of our response time predictions ranges from 10% to 16% on these applications, and our models generalize well to workloads very different from those used for calibration. We apply our technique to the challenging problem of predicting the impact of application consolidation on transaction response times. We calibrate models of two testbed applications running on dedicated machines, then use the models to predict their performance when they run together on a shared machine and serve very different workloads. Our predictions are accurate to within 4% to 14%. Existing approaches to consolidation decision support predict post-consolidation resource utilizations. Our method allows application-level performance to guide consolidation decisions.

The theory of deadlock avoidance via discrete control

Deadlock in multithreaded programs is an increasingly important problem as ubiquitous multicore architectures force parallelization upon an ever wider range of software. This paper presents a theoretical foundation for dynamic deadlock avoidance in concurrent programs that employ conventional mutual exclusion and synchronization primitives (e.g., multithreaded C/Pthreads programs). Beginning with control flow graphs extracted from program source code, we construct a formal model of the program and then apply Discrete Control Theory to automatically synthesize deadlock-avoidance control logic that is implemented by program instrumentation. At run time, the control logic avoids deadlocks by postponing lock acquisitions. Discrete Control Theory guarantees that the program instrumented with our synthesized control logic cannot deadlock. Our method furthermore guarantees that the control logic is maximally permissive: it postpones lock acquisitions only when necessary to prevent deadlocks, and therefore permits maximal runtime concurrency. Our prototype for C/Pthreads scales to real software including Apache, OpenLDAP, and two kinds of benchmarks, automatically avoiding both injected and naturally occurring deadlocks while imposing modest runtime overheads.

Eliminating Concurrency Bugs with Control Engineering

In the multicore era, concurrency bugs threaten to reduce programmer productivity, impair software safety, and erode end-user value. Control engineering can eliminate concurrency bugs by constraining software behavior, preventing runtime failures, and offloading onerous burdens from human programmers onto automatically synthesized control logic.

Generalized knapsack solvers for multi-unit combinatorial auctions: analysis and application to computational resource allocation

The problem of allocating discrete computational resources motivates interest in general multi-unit combinatorial exchanges. This paper considers the problem of computing optimal (surplus-maximizing) allocations, assuming unrestricted quasi-linear preferences. We present a solver whose pseudo-polynomial time and memory requirements are linear in three of four natural measures of problem size: number of agents, length of bids, and units of each resource. In applications where the number of resource types is inherently a small constant, e.g., computational resource allocation, such a solver offers advantages over more elaborate approaches developed for high-dimensional problems. We also describe the deep connection between auction winner determination problems and generalized knapsack problems, which has received remarkably little attention in the literature. This connection leads directly to pseudo-polynomial solvers, informs solver benchmarking by exploiting extensive research on hard knapsack problems, and allows E-Commerce research to leverage a large and mature body of literature.

Failure-atomic msync(): a simple and efficient mechanism for preserving the integrity of durable data

Preserving the integrity of application data across updates is difficult if power outages and system crashes may occur during updates. Existing approaches such as relational databases and transactional key-value stores restrict programming flexibility by mandating narrow data access interfaces. We have designed, implemented, and evaluated an approach that strengthens the semantics of a standard operating system primitive while maintaining conceptual simplicity and supporting highly flexible programming: Failureatomic msync() commits changes to a memory-mapped file atomically, even in the presence of failures. Our Linux implementation of failure-atomic msync() has preserved application data integrity across hundreds of whole-machine power interruptions and exhibits good microbenchmark performance on both spinning disks and solid-state storage. Failure-atomic msync() supports higher layers of fully general programming abstraction, e.g., a persistent heap that easily slips beneath the C++ Standard Template Library. An STL <map> built atop failure-atomic msync() outperforms several local key-value stores that support transactional updates. We integrated failure-atomic msync() into the Kyoto Tycoon key-value server by modifying exactly one line of code; our modified server reduces response times by 26--43% compared to Tycoons existing transaction support while providing the same data integrity guarantees. Compared to a Tycoon server setup that makes almost no I/O (and therefore provides no support for data durability and integrity over failures), failure-atomic msync() incurs a three-fold response time increase on a fast Flash-based SSD---an acceptable cost of data reliability for many.

Discrete control for safe execution of IT automation workflows

As information technology (IT) administration becomes increasingly complex, workflow technologies are gaining popularity for IT automation. Writing correct workflow programs is notoriously difficult. Although static analysis tools are available, fixing defects remains manual and error-prone. This paper applies discrete control theory to IT automation workflows. Discrete control detects flaws in workflows just as static analysis does, and more importantly it also allows safe execution of flawed workflows by dynamically avoiding run-time failures. Our approach can guarantee compliance with certain requirements and can partially decouple requirements from software, reducing the need to modify the latter if the former change. We have implemented a discrete control module for a real IT automation system. Experiments with workflows from a real production system and with randomly generated workflows show that our approach scales to workflows of practical size.

Gadara nets: Modeling and analyzing lock allocation for deadlock avoidance in multithreaded software

Deadlock avoidance in shared-memory multithreaded programs is receiving increased attention as multicore architectures and parallel programming are becoming more prevalent. In our on-going project, called Gadara, the objective is to control the execution of multithreaded programs in order to avoid deadlocks by using techniques from discrete-event control theory. In this project, Petri nets are employed to model parallel programs. This paper formally defines the class of Petri nets that emerges from modeling multithreaded programs, called Gadara nets. Gadara nets are related to, but different from, other classes of nets that have been characterized in deadlock analysis of manufacturing systems. The contributions of this paper include: (i) formal definition of Gadara nets and of controlled Gadara nets; (ii) a behavioral analysis of Gadara nets for liveness and reversibility using siphons; and (iii) identification of a convexity-type property for the set of live markings.

Concurrency bugs in multithreaded software: modeling and analysis using Petri nets

In this paper, we apply discrete-event system techniques to model and analyze the execution of concurrent software. The problem of interest is deadlock avoidance in shared-memory multithreaded programs. We employ Petri nets to systematically model multithreaded programs with lock acquisition and release operations. We define a new class of Petri nets, called Gadara nets, that arises from this modeling process. We investigate a set of important properties of Gadara nets, such as liveness, reversibility, and linear separability. We propose efficient algorithms for the verification of liveness of Gadara nets, and report experimental results on their performance. We also present modeling examples of real-world programs. The results in this paper lay the foundations for the development of effective control synthesis algorithms for Gadara nets.

Eliminating Concurrency Bugs in Multithreaded Software: A New Approach Based on Discrete-Event Control

Computer hardware is moving from uniprocessor to multicore architectures. One problem arising in this evolution is that only parallel software can exploit the full performance potential of multicore architectures, and parallel software is far harder to write than conventional serial software. One important class of failures arising in parallel software is circular-wait deadlock in multithreaded programs. In our ongoing Gadara project, we use a special class of Petri nets, called Gadara nets, to systematically model multithreaded programs with lock allocation and release operations. In this paper, we propose an efficient optimal control synthesis methodology for ordinary Gadara nets that exploits the structural properties of Gadara nets via siphon analysis. Optimality in this context refers to the elimination of deadlocks in the program with minimally restrictive control logic. We formally establish a set of important properties of the proposed control synthesis methodology, and show that our algorithms never synthesize redundant control logic. We conduct experiments to evaluate the efficiency and scalability of the proposed methodology, and discuss the application of our results to real-world concurrent software.

Supervisory Control of Software Execution for Failure Avoidance: Experience from the Gadara Project

Abstract We discuss our experience in the Gadara project, whose objective is to control the execution of software to avoid potential failures using discrete-event control techniques. We summarize our accomplishments so far and discuss future challenges. After initial work on safety of workflow scripts via supervisory control techniques, we have focused our efforts on deadlock avoidance in multithreaded C programs that use locking primitives to control access to shared data. We describe how we automatically construct automata models of workflows and Petri net models of concurrent programs. In the case of multithreaded C programs, the resulting models characterize a new class of resource-allocation Petri nets called Gadara nets. These nets enjoy structural properties that facilitate the synthesis of liveness-enforcing control policies that are maximally-permissive. We describe our strategy for run-time implementation of these control policies, especially by a technique known as code instrumentation. It is hoped that the lessons learned so far in the Gadara project will be useful in other application areas and will suggest avenues for future theoretical investigations.