Rachel Tzoref

Deadlocks: From Exhibiting to Healing

Deadlocks are possibly the best known bug pattern in computer systems in general; certainly they are the best known in concurrent programming. Numerous articles, some dating back more than 40 years, have been dedicated to the questions of how to design deadlock free programs, how to statically or dynamically detect possible deadlocks, how to avoid deadlocks at runtime, and how to resolve deadlocks once they happen. We start the paper with an investigation on how to exhibit potential deadlocks. Exhibiting deadlocks is very useful in testing, as verifying if a potential deadlock can actually happen is a time-consuming debugging activity. There was recently some very interesting research in this direction; however, we believe our approach is more practical, has no scaling issues, and in fact is already industry-ready. n nThe second contribution of our paper is in the area of healing multi-threaded programs so they do not get into deadlocks. This is an entirely new approach, which is very different from the approaches in the literature that were meant for multi-process scenarios and are not suitable (and indeed not used) in multithreaded programming. While the basic ideas are fairly simple, the details here are very important as any mistake is liable to actually create new deadlocks. The paper describes the basic healing idea and its limitations, the pitfalls and how to overcome them, and experimental results.

Healing data races on-the-fly

Testing of concurrent software is extremely difficult. Despite all the progress in the testing and verification technology, concurrent bugs, the most common of which are deadlocks and races, make it to the field. This paper describes a set of techniques, implemented in a tool called ConTest, allowing concurrent programs to self-heal at run-time. Concurrent bugs have the very desirable property for healing that some of the interleaving produce correct results while in others bugs manifest. Healing concurrency problems is about limiting, or changing the probability of interleaving, such that bugs will be seen less. When healing concurrent programs, if a deadlock does not result from limiting the interleaving, we are sure that the result of the healed program could have been in the original program and therefore no new functional bug has been introduced. In this initial work which deals with different types of data races, we suggest three types of healing mechanisms: (1) changing the probability of interleaving by introducing sleep or yield statements or by changing thread priorities, (2) removing interleaving using synchronisation commands like locking and unlocking certain mutexes or waits and notifies, and (3) removing the result of bad interleaving by replacing the value of variables by the one that should have been taken. We also classify races according to the relevant healing strategies to apply.

Instrumenting where it hurts: an automatic concurrent debugging technique

As concurrent and distributive applications are becoming more common and debugging such applications is very difficult, practical tools for automatic debugging of concurrent applications are in demand. In previous work, we applied automatic debugging to noise-based testing of concurrent programs. The idea of noise-based testing is to increase the probability of observing the bugs by adding, using instrumentation, timing noise to the execution of the program. The technique of finding a small subset of points that causes the bug to manifest can be used as an automatic debugging technique. Previously, we showed that Delta Debugging can be used to pinpoint the bug location on some small programs.n In the work reported in this paper, we create and evaluate two algorithms for automatically pinpointing program locations that are in the vicinity of the bugs on a number of industrial programs. We discovered that the Delta Debugging algorithms do not scale due to the non-monotonic nature of the concurrent debugging problem. Instead we decided to try a machine learning feature selection algorithm. The idea is to consider each instrumentation point as a feature, execute the program many times with different instrumentations, and correlate the features (instrumentation points) with the executions in which the bug was revealed. This idea works very well when the bug is very hard to reveal using instrumentation, correlating to the case when a very specific timing window is needed to reveal the bug. However, in the more common case, when the bugs are easy to find using instrumentation points ranked high by the feature selection algorithm is not high enough. We show that for these cases, the important value is not the absolute value of the evaluation of the feature but the derivative of that value along the program execution path.n As a number of groups expressed interest in this research, we built an open infrastructure for automatic debugging algorithms for concurrent applications, based on noise injection based concurrent testing using instrumentation. The infrastructure is described in this paper.

Automatic refinement and vacuity detection for symbolic trajectory evaluation

Symbolic Trajectory Evaluation (STE) is a powerful technique for model checking. It is based on 3-valued symbolic simulation, using 0,1 and X (”unknown”). The X value is used to abstract away parts of the circuit. The abstraction is derived from the users specification. Currently the process of abstraction and refinement in STE is performed manually. This paper presents an automatic refinement technique for STE. The technique is based on a clever selection of constraints that are added to the specification so that on the one hand the semantics of the original specification is preserved, and on the other hand, the part of the state space in which the ”unknown” result is received is significantly decreased or totally eliminated. In addition, this paper raises the problem of vacuity of passed and failed specifications. This problem was never discussed in the framework of STE. We describe when an STE specification may vacuously pass or fail, and propose a method for vacuity detection in STE.

An Optimized Symbolic Bounded Model Checking Engine

It has been shown that bounded model checking using a SAT solver can solve many verification problems that would cause BDD based symbolic model checking engines to explode. However, no single algorithmic solution has proven to be totally superior in resolving all types of model checking problems. We present an optimized bounded model checker based on BDDs and describe the advantages and drawbacks of this model checker as compared to BDD-based symbolic model checking and SAT-based model checking. We show that, in some cases, this engine solves verification problems that could not be solved by other methods.

Supporting SAT based BMC on Finite Path Models

The standard translation of a Bounded Model Checking (BMC) instance into a satisfiability problem, (a.k.a SAT), might produce misleading results in the case when the model under verification contains finite paths. Models with finite paths might be produced unknowingly when using modern verification languages such as PSL-Sugar [Property Specification Language: Reference Manual. Version 1.1, Accellera, June 2004]. Specifically, the use of language constructs such as restrict, assume etc. might lead to such models. Thus the user may receive misleading results from SAT based tools. In this paper we describe in what circumstances the finite path problem occurs and present an improved translation of the BMC problem into a SAT instance. The new translation does not suffer from the discussed shortcoming. Our translation is only slightly longer then the usual one introducing one extra Boolean variable in the model. We also show that this translation may improve the SAT solver runtime even for models without finite paths.

Automatic Debugging of Concurrent Programs through Active Sampling of Low Dimensional Random Projections

Concurrent computer programs are fast becoming prevalent in many critical applications. Unfortunately, these programs are especially difficult to test and debug. Recently, it has been suggested that injecting random timing noise into many points within a program can assist in eliciting bugs within the program. Upon eliciting the bug, it is necessary to identify a minimal set of points that indicate the source of the bug to the programmer. In this paper, we pose this problem as an active feature selection problem. We propose an algorithm called the iterative group sampling algorithm that iteratively samples a lower dimensional projection of the program space and identifies candidate relevant points. We analyze the convergence properties of this algorithm. We test the proposed algorithm on several real-world programs and show its superior performance. Finally, we show the algorithms performance on a large concurrent program.