Kaiyuan Wang

Automated Test Generation and Mutation Testing for Alloy

We present two novel approaches for automated testing of models written in Alloy – a well-known declarative, first-order language that is supported by a fully automatic SAT-based analysis engine. The first approach introduces automated test generation for Alloy and is embodied by three techniques that create test suites in the traditional spirit of black-box, white-box, and mutation-based testing. The second approach introduces mutation testing for Alloy and defines how to create mutants of Alloy models, compute mutation testing results, and check for equivalent mutants using SAT. The two approaches build on the theoretical foundation defined previously by our AUnit framework, which introduced the idea of unit testing for Alloy in the spirit of unit testing for imperative languages. While test generation and mutation testing are heavily studied problems with many solutions in the context of imperative languages, the key novelty of our work is to introduce and address these problems for the declarative programming paradigm, specifically for the Alloy language. Experimental results using several Alloy subjects, including those with real faults, demonstrate the efficacy of our framework.

MuAlloy: a mutation testing framework for alloy

Creating models of software systems and analyzing the models helps develop more reliable systems. A well-known software modeling tool-set is embodied by the declarative language Alloy and its automatic SAT-based analyzer. Recent work introduced a novel approach to testing Alloy models to validate their correctness in the spirit of traditional software testing: AUnit defined the foundations of testing (unit tests, test execution, and model coverage) for Alloy, and MuAlloy defined mutation testing (mutation operators, mutant generation, and equivalent mutant checking) for Alloy. This tool paper describes our Java implementation of MuAlloy, which is a command-line tool that we released as an open-source project on GitHub. Our experimental results show that MuAlloy is efficient and practical. The demo video for MuAlloy can be found at https://youtu.be/3lvnQKiLcLE.

Systematic Generation of Non-equivalent Expressions for Relational Algebra

Relational algebra forms the semantic foundation in multiple domains, e.g., Alloy models, OCL constraints, UML metamodels, and SQL queries. Synthesis and repair techniques in such domains require an efficient procedure to generate (non-equivalent) expressions subject to relational constraints, e.g., the types of sets and relations, their cardinality, size of expressions, maximum arity of the intermediate expressions, etc. This paper introduces the first generator for relational expressions that are non-equivalent with respect to the semantics of relational algebra. We present the algorithms that define our generator, its embodiment based on the Alloy tool-set, and an experimental evaluation to show the effectiveness of its non-equivalent generation for a variety of problems with relational constraints.

Solver-Based Sketching of Alloy Models Using Test Valuations

We introduce ASketch, the first framework for sketching models in the Alloy language. The Alloy Analyzer is a SAT-based constraint solver that allows users to create valuations for relations with respect to given constraints and bound on the universe of discourse. Alloy users routinely use the valuations to validate their models: enumerate some valuations and inspect them to detect underconstraints or overconstraints. Our key insight is that valid and invalid valuations enable sketching Alloy models where the user writes a partial model with holes and provides some valuations, and the sketching infrastructure completes the model by synthesizing Alloy fragments for the holes.

Automated model repair for Alloy

Automated program repair is an active research area. However, existing research focuses mostly on imperative code, e.g. in Java. In this paper, we study the problem of repairing declarative models in Alloy - a first order relational logic with transitive closure. We introduce ARepair, the first technique for repairing Alloy models. ARepair follows the spirit of traditional automated program repair techniques. Specifically, ARepair takes as input a faulty Alloy model and a test suite that contains some failing test, and outputs a repaired model that is correct with respect to the given tests. ARepair integrates ideas from mutation testing and program synthesis to provide an effective solution for repairing Alloy models. The experimental results show that ARepair can fix 28 out of 38 real-world faulty models we collected.

JPR: Replaying JPF Traces Using Standard JVM

Java PathFinder (JPF) is a backtrackable Java Virtual Machine (JVM), which is implemented in Java and runs on a standard JVM (e.g., Oracle HotSpot). Thus, a JPF developer can use off-the- shelf Java debuggers (e.g., jdb) when debugging code that makes up JPF. JPF explores all non-deterministic executions of a given target program and monitors for property violations. To facilitate debugging of the target program, JPF can capture and replay the execution trace that leads to a property violation. While the deterministic replay is invaluable, the replay with JPF does not allow the developer to attach an off-the-shelf Java debugger to the target program (e.g., step through the application code, set breakpoints, etc.). We present a technique, dubbed JPR, to improve the debugging experience of the JPF captured traces by migrating the JPF traces to a new format that can be executed using the standard JVM. JPR annotates each JPF trace, during the capture phase, with ex- tra data (e.g., instruction index, instruction count, etc.); the an- notated trace is then used to instrument Java bytecode to enforce the same execution trace on a standard JVM. JPR is compatible with various optimizations, e.g., state matching and partial-order reduction. We evaluated JPR on all multi-threaded Java pro- grams in the official JPF distribution. Our results show that JPR successfully replayed all JPF traces on the standard JVM with reasonable overhead during both recording and replaying.

Repairing Intricate Faults in Code Using Machine Learning and Path Exploration

Debugging remains costly and tedious, especially for code that performs intricate operations that are conceptually complex to reason about. We present MLR, a novel approach for repairing faults in such operations, specifically in the context of complex data structures. Our focus is on faults in conditional statements. Our insight is that an integrated approach based on machine learning and systematic path exploration can provide effective repairs. MLR mines the data-spectra of the passing and failing executions of conditional branches to prune the search space for repair and generate patches that are likely valid beyond the existing test-suite. We apply MLR to repair faults in small but complex data structure subjects to demonstrate its efficacy. Experimental results show that MLR has the potential to repair this fault class more effectively than state-of-the-art repair tools.