Shmuel Ur

Towards a framework and a benchmark for testing tools for multi-threaded programs

Multi‐threaded code is becoming very common, both on the server side, and very recently for personal computers as well. Consequently, looking for intermittent bugs is a problem that is receiving more and more attention. As there is no silver bullet, research focuses on a variety of partial solutions. We outline a road map for combining the research within the different disciplines of testing multi‐threaded programs and for evaluating the quality of this research. We have three main goals. First, to create a benchmark that can be used to evaluate different solutions. Second, to create a framework with open application programming interfaces that enables the combination of techniques in the multi‐threading domain. Third, to create a focus for the research in this area around which a community of people who try to solve similar problems with different techniques can congregate. We have started creating such a benchmark and describe the lessons learned in the process. The framework will enable technology developers, for example, developers of race detection algorithms, to concentrate on their components and use other ready made components (e.g. an instrumentor) to create a testing solution. Copyright

Deadlocks: From Exhibiting to Healing

Deadlocks are possibly the best known bug pattern in computer systems in general; certainly they are the best known in concurrent programming. Numerous articles, some dating back more than 40 years, have been dedicated to the questions of how to design deadlock free programs, how to statically or dynamically detect possible deadlocks, how to avoid deadlocks at runtime, and how to resolve deadlocks once they happen. We start the paper with an investigation on how to exhibit potential deadlocks. Exhibiting deadlocks is very useful in testing, as verifying if a potential deadlock can actually happen is a time-consuming debugging activity. There was recently some very interesting research in this direction; however, we believe our approach is more practical, has no scaling issues, and in fact is already industry-ready. The second contribution of our paper is in the area of healing multi-threaded programs so they do not get into deadlocks. This is an entirely new approach, which is very different from the approaches in the literature that were meant for multi-process scenarios and are not suitable (and indeed not used) in multithreaded programming. While the basic ideas are fairly simple, the details here are very important as any mistake is liable to actually create new deadlocks. The paper describes the basic healing idea and its limitations, the pitfalls and how to overcome them, and experimental results.

Compiling a benchmark of documented multi-threaded bugs

Summary form only given. Testing multithreaded, concurrent, or distributed programs is acknowledged to be a very difficult task. We decided to create a benchmark of programs containing documented multithreaded bugs that can be used in the development of testing tool for the domain. In order to augment the benchmark with a sizable number of programs, we assigned students in a software testing class to write buggy multithreaded Java programs and document the bugs. This paper documents this experiment. We explain the task that was given to the students, go over the bugs that they put into the programs both intentionally and unintentionally, and show our findings. We believe this part of the benchmark shows typical programming practices, including bugs, of novice programmers. In grading the assignments, we used our technologies to look for undocumented bugs. In addition to finding many undocumented bugs, which was not surprising given that writing correct multithreaded code is difficult, we also found a number of bugs in our tools. We think this is a good indication of the expected utility of the benchmark for multithreaded testing tool creators.

Advanced Analysis Techniques for Cross-Product Coverage

Coverage analysis is used to monitor the quality of the verification process. Reports provided by coverage tools help users identify areas in the design that have not been adequately tested. Becaus...

A Concurrency Testing Tool and Its Plug-Ins for Dynamic Analysis and Runtime Healing

This paper presents a tool for concurrency testing (abbreviated as ConTest) and some of its extensions. The extensions (called plug-ins in this paper) are implemented through the listener architecture of ConTest. Two plug-ins for runtime detection of common concurrent bugs are presented--the first (Eraser+) is able to detect data races while the second (AtomRace) is able to detect not only data races but also more general bugs caused by violation of atomicity presumptions. A third plug-in presented in this paper is designed to hide bugs that made it into the field so that when problems are detected they can be circumvented. Several experiments demonstrate the capabilities of these plug-ins.

Testing of concurrent programs using genetic algorithms

Noise injection disturbs the scheduling of program threads in order to increase the probability that more of their different legal interleavings occur during the testing process. However, there exist many different types of noise heuristics with many different parameters that are not easy to set such that noise injection is really efficient. In this paper, we propose a new way of using genetic algorithms to search for suitable types of noise heuristics and their parameters. This task is formalized as the test and noise configuration search problem in the paper, followed by a discussion of how to represent instances of this problem for genetic algorithms, which objectives functions to use, as well as parameter tuning of genetic algorithms when solving the problem. The proposed approach is evaluated on a set of benchmarks, showing that it provides significantly better results than the so far preferred random noise injection.

Toward Automatic Concurrent Debugging Via Minimal Program Mutant Generation with AspectJ

Debugging is one of the most time-consuming activities in program design. Work on automatic debugging has received a great deal of attention and there are a number of symposiums dedicated to this field. Automatic debugging is usually invoked when a test fails in one situation, but succeeds in another. For example, a test fails in one version of the program (or scheduler), but succeeds in another. Automatic debugging searches for the smallest difference that causes the failure. This is very useful when working to identify and fix the root cause of the bug. A new testing method instruments concurrent programs with schedule-modifying instructions to reveal concurrent bugs. This method is designed to increase the probability of concurrent bugs (such as races, deadlocks) appearing. This paper discusses integrating this new testing technology with automatic debugging. Instead of just showing that a bug exists, we can pinpoint its location by finding the minimal set of instrumentations that reveal the bug. In addition to explaining a methodology for this integration, we show an AspectJ-based implementation. We discuss the implementation in detail as it both demonstrates the advantage of the adaptability of open source tools and how our specific change can be used for other testing tools.

A probabilistic alternative to regression suites

Automated regression suites are essential in developing large applications, while maintaining reasonable quality and timetables. The main argument against the automation of regression suites, in addition to the cost of creation and maintenance, is the observation that if you run the same test many times, it becomes increasingly less likely to find bugs. To alleviate such problems, a new regression suite practice, using random test generators to create regression suites on-the-fly, is becoming more common. In this practice, instead of maintaining tests, we generate test suites on-the-fly by choosing several specifications and generating a number of tests from each specification. We describe techniques for optimizing random generated test suites. We first show how the set cover greedy algorithms, commonly used for selecting tests for regression suites, may be adapted to selecting specifications for randomly generated regression suites. We then introduce a new class of greedy algorithms, referred to as future-aware greedy algorithms. The algorithms are computationally efficient and generate more effective regression suites.

Improving throughput via slowdowns

Many service-oriented systems are not well equipped to guarantee that service time is optimized. We have specifically examined two industrial systems which implement service-oriented architectures in real, field environments. We discovered that both were not engineered to properly address surges in service request rate. In the absence of an integral solution, it is difficult and costly to (re-) engineer such a solution in the field. The challenge faced by this study was to deliver a low cost solution, without re-engineering the target systems. This paper introduces such a generic solution. The solution slows-down some components to deliver improvement in request service time. It was implemented, tested, and successfully applied to two industrial systems with no need to modify their logic or architecture. Experiments with those systems exhibited significant improvement in performance. These results have validated our solution and its industrial applicability across systems and environments.

Testing concurrent software

This short two hour tutorial tries to teach practical techniques for testing, review and debugging of concurrent software, based on the Dr. Ur’s research and practical industrial experience in the last ten years.