Kari Kähkönen

Improving Dynamic Partial Order Reductions for Concolic Testing

Testing multi-threaded programs is hard due to the state explosion problem arising from the different interleavings of concurrent operations. The dynamic partial order reduction (DPOR) algorithm by Flanagan and Godefroid is one solution to reducing this problem. We present a modification to this algorithm that allows it to exploit the commutativity of read operations and provide further reduction. To enable testing of multi-threaded programs that also take input we show that it is possible to combine DPOR with concolic testing. We have implemented our modified DPOR algorithm in the LCT concolic testing tool. We have also implemented the sleep set algorithm, which can be used along with DPOR to provide further reduction. As the LCT tool was designed for distributed use we have modified the sleep set algorithm for use in a distributed testing client-server setting.

The LIME Interface Specification Language and Runtime Monitoring Tool

This paper describes an interface specification language designed in the LIME project (LIME ISL) and the supporting runtime monitoring tool. The interface specification language is tailored for the Java programming language and supports two kinds of specifications: (i) call specifications that specify requirements for the allowed call sequences to a Java object instance and (ii) return specifications that specify the allowed behaviors of the Java object instance. Both the call and return specifications can be expressed with Java annotations in several different ways: as past time LTL formulas, as (safety) future LTL formulas, as regular expressions, and as nondeterministic finite automata. We also describe the supporting LIME interface monitoring tool which is an open source implementation of runtime monitoring for the interface specifications implemented using AspectJ.

Using unfoldings in automated testing of multithreaded programs

In multithreaded programs both environment input data and the nondeterministic interleavings of concurrent events can affect the behavior of the program. One approach to systematically explore the nondeterminism caused by input data is dynamic symbolic execution. For testing multithreaded programs we present a new approach that combines dynamic symbolic execution with unfoldings, a method originally developed for Petri nets but also applied to many other models of concurrency. We provide an experimental comparison of our new approach with existing algorithms combining dynamic symbolic execution and partial-order reductions and show that the new algorithm can explore the reachable control states of each thread with a significantly smaller number of test runs. In some cases the reduction to the number of test runs can be even exponential allowing programs with long test executions or hard-to-solve constrains generated by symbolic execution to be tested more efficiently.

Testing Multithreaded Programs with Contextual Unfoldings and Dynamic Symbolic Execution

Systematic testing of multithreaded programs is challenging as possible execution paths depend not only on input values but also on how threads interleave their executions. As the number of all interleavings is typically too large to explore exhaustively, ways to avoid testing irrelevant interleavings are needed. We have previously presented a testing approach that is based on dynamic symbolic execution to handle input values and on Petri net unfoldings to succinctly represent how threads can interleave their executions. In this paper we improve this approach by using contextual nets instead of ordinary nets. This way, the interleavings of threads can be represented even more succinctly which allows the local states of threads to be covered with less test executions. Furthermore, the new approach makes it possible to test programs that create threads dynamically which was a limitation of the previous approach. We have implemented our improved algorithm in a testing tool for Java programs. Experiments demonstrate that the new approach can significantly reduce the number of needed tests to cover all the reachable local states of threads while keeping additional overhead required by the algorithm small.

Experimental comparison of concolic and random testing for java card applets

Concolic testing is a method for test input generation where a given program is executed both concretely and symbolically at the same time. This paper introduces the LIME Concolic Tester (LCT), an open source concolic testing tool for sequential Java programs. It discusses the design choices behind LCT as well as its use in automated unit test generation for the JUnit testing framework. As the main experimental contribution we report on an empirical evaluation of LCT for testing smart card Java applets. In particular, we focus on the problem of differential testing, where a Java class implementation is tested against a reference implementation. Two different concolic unit test generation approaches are presented and their effectiveness is compared with random testing. The experiments show that concolic testing is able to find significantly more bugs than random testing in the testing domain at hand.

Unfolding based automated testing of multithreaded programs

In multithreaded programs both environment input data and the nondeterministic interleavings of concurrent events can affect the behavior of the program. One approach to systematically explore the nondeterminism caused by input data is dynamic symbolic execution. For testing multithreaded programs we present a new approach that combines dynamic symbolic execution with unfoldings, a method originally developed for Petri nets but also applied to many other models of concurrency. We provide an experimental comparison of our new approach with existing algorithms combining dynamic symbolic execution and partial order reductions and show that the new algorithm can explore the reachable control states of each thread with a significantly smaller number of test runs. In some cases the reduction to the number of test runs can be even exponential allowing programs with long test executions or hard-to-solve constraints generated by symbolic execution to be tested more efficiently. In addition we show that our algorithm generates a structure describing different interleavings from which deadlocks can be detected efficiently as well.

Lightweight State Capturing for Automated Testing of Multithreaded Programs

We present a lightweight approach to capture abstract state information that can be used to avoid testing redundant interleavings of multithreaded programs. Our approach is based on modeling states that are observed during the test executions as a Petri net. This model is then used to determine if some execution paths lead to an already explored state. In such cases exploring execution paths from the same state multiple times can be avoided. Our approach does not capture the complete global states of programs but instead it relies on particular commutativity of transitions to determine if they lead to already known abstract states. We have combined this lightweight state capture technique with a dynamic symbolic execution based approach to systematically test multithreaded programs. Experiments show that even without complete state information, the lightweight state capturing technique can sometimes reduce the number of redundant test executions substantially.

Unfolding Based Minimal Test Suites for Testing Multithreaded Programs

This paper focuses on the problem of computing the minimal test suite for a terminating multithreaded program that covers all its executable statements. We have in previous work shown how to use unfoldings to capture the true concurrency semantics of multithreaded programs and to generate test cases for it. In this paper we rely on this earlier work and show how the unfolding can be used to generate the minimal test suite that covers all the executable statements of the program. The problem of generating such a minimal test suite is shown to be NP-complete in the size of the unfolding, and as a side result, covering executable transitions of any terminating safe Petri net is also NP-complete in the size of its unfolding. We propose SMT-encodings to these problems and give initial results on applying this encoding to compute the minimal test suite for several benchmarks.

Minimizing Test Suites with Unfoldings of Multithreaded Programs

This article focuses on computing minimal test suites for multithreaded programs. Based on previous work on test case generation for multithreaded programs using unfoldings, this article shows how this unfolding can be used to generate minimal test suites covering all local states of the program. Generating such minimal test suites is shown to be NP-complete in the size of the unfolding. We propose an SMT encoding for this problem and two methods based on heuristics which only approximate the solution, but scale better in practice. Finally, we apply our methods to compute the minimal test suites for several benchmarks.