Susumu Tokumoto

MuVM: Higher Order Mutation Analysis Virtual Machine for C

Mutation analysis is a method for evaluating the effectiveness of a test suite by seeding faults artificially and measuring the fraction of seeded faults detected by the test suite. The major limitation of mutation analysis is its lengthy execution time because it involves generating, compiling and running large numbers of mutated programs, called mutants. Our tool MuVM achieves a significant runtime improvement by performing higher order mutation analysis using four techniques, meta mutation, mutation on virtual machine, higher order split-stream execution, and online adaptation technique. In order to obtain the same behavior as mutating the source code directly, meta mutation preserves the mutation location information which may potentially be lost during bit code compilation and optimization. Mutation on a virtual machine reduces the compilation and testing cost by compiling a program once and invoking a process once. Higher order split-stream execution also reduces the testing cost by executing common parts of the mutants together and splitting the execution at a seeded fault. Online adaptation technique reduces the number of generated mutants by omitting infeasible mutants. Our comparative experiments indicate that our tool is significantly superior to an existing tool, an existing technique (mutation schema generation), and no-split-stream execution in higher order mutation.

FSX: fine-grained incremental unit test generation for C/C++ programs

Automated unit test generation bears the promise of significantly reducing test cost and hence improving software quality. However, the maintenance cost of the automatically generated tests presents a significant barrier to adoption of this technology. To address this challenge, we propose a novel technique for automated and fine-grained incremental generation of unit tests through minimal augmentation of an existing test suite. The technique uses iterative, incremental refinement of test-drivers and symbolic execution, guided by a diagnostics engine. The diagnostics engine works off a novel precise and efficient byte-level dynamic dependence analysis built using Reduced Ordered Binary Decision Diagrams (ROBDDs). We present a tool FSX implementing this technique and evaluate it under two practical use-cases of incremental unit test generation, on five revisions of the open-source software iPerf, as well as on 3 large subjects, comprising more than 60 thousand lines of code, from in-house commercial network products. The evaluation shows that FSX can generate high-quality unit tests on large industrial software while minimizing the maintenance cost of the overall test-suite.

Enhancing Symbolic Execution to Test the Compatibility of Re-engineered Industrial Software

After a legacy system is re-engineered, it is important to perform compatibility testing so as to identify the difference and reduce the introduced bugs. We can first apply symbolic execution to obtain an exhaustive set of test cases, then use them to check the compatibility of the old system and the new old. However, the path explosion problem of symbolic execution makes it difficult to work on realistic non-trivial applications. We show in this paper how to enhance symbolic execution, e.g. with extra constraints, path cutting, variable grouping, and test case selection, to successfully test the compatibility of an SMTP library (used in embedded systems) with around 20K lines of code. Our experience indicates that these enhancements are essential to apply symbolic execution on realistic industrial applications.

KLOVER: Automatic Test Generation for C and C Programs, Using Symbolic Execution

Fujitsu researchers have developed a methodology to automate testing of industrial-strength embedded software implemented in C or C++. The methodology’s core is a program analysis technique called symbolic execution, which the researchers have customized to automate testing. The methodology generates unit-level tests, greatly reducing test generation time and cost while providing excellent test coverage.

Semi-automatic Incompatibility Localization for Re-engineered Industrial Software

After a legacy system is re-engineered, it is important to perform compatibility testing so as to identify the differences and reduce the introduced bugs. We can first apply symbolic execution to obtain an exhaustive set of test cases, then use them to check the compatibility of the old system and the new one. However there may be a lot of failed test cases which are a mix of erroneous and allowable incompatibilities. To locate the causes of failures detected during the testing, we apply multiple statistical bug localization techniques. We are able to localize 90% of the incompatibilities in 10% of the code for an industrial application with around 20k lines by Tarantula. And we identify the characteristics of failure causes which are difficult to be detected by statistical bug localization.

Elixir: an automated repair tool for Java programs

Object-oriented (OO) languages, by design, make heavy use of method invocations (MI). Unsurprisingly, a large fraction of OO-program bug patches also involves method invocations. However, current program repair techniques incorporate MIs in very limited ways, ostensibly to avoid searching the huge repair space that method invocations afford. To address this challenge, in previous work, we proposed a generate-and-validate repair technique which can effectively synthesize patches from a repair space rich in method invocation expressions, by using a machine-learned model to rank the space of concrete repairs. In this paper, we describe the tool Elixir that instantiates this technique for the repair of Java programs. We describe the architecture, user-interface, and salient features of Elixir, and specific use-cases it can be applied in. We also report on our efforts towards practical deployment of Elixir within our organization, including the initial results of a trial of Elixir on a project of interest to potential customers. A video demonstrating Elixir is available at: https://elixir-tool.github.io/demo-video.html

FSX: a tool for fine-grained incremental unit test generation for C/C++ programs

Automated unit test generation bears the promise of significantly reducing test cost and hence improving software quality. However, the maintenance cost of the automatically generated tests presents a significant barrier to adoption of this technology. To address this challenge, in previous work, we proposed a novel technique for automated and fine-grained incremental generation of unit tests through minimal augmentation of an existing test suite. In this paper we describe a tool FSX, implementing this technique. We describe the architecture, user-interface, and salient features of FSX, and specific practical use-cases of its technology. We also report on a real, large-scale deployment of FSX, as a practical validation of the underlying research contribution and of automated test generation research in general.

Model-based test case generation using symbolic execution

In this paper, we present a test case generation method in which test cases are generated from Excel-based functional specications, called error-check/update specications. This method has the following two characteristics. 1) Logical structures and constraint conditions in error-check/update specications are translated into Java code. Then the trans- lated Java code is executed symbolically. A solution found by solving combinations of the constraint conditions, during the symbolic execution can serve as a test case for the Excel- based specication. 2)We present test case selection method to extract a suitable test-suite from the large number of test cases typically generated by the above symbolic execution. Our test case selection is based on a path condition-based test case selection criterion, proposed in our previous work, in which the selected test cases covers all the atomic conditions of path conditions collected during symbolic execution. In this paper, we extend this method with classication of test cases and priority of variable values with the aim of improv- ing the quality of the selected test-suite. We implemented the test case generation method in our tool and evaluated this method with Excel-based test specications for a real- world nancial system.