Eun-Hye Choi

Model checking class specifications for Web applications

This paper proposes an approach for verifying class specifications of Web applications using model checking. We first present a method to model a dynamic behavior of a Web application from a class specification. We next propose two methods to verify consistencies of the class specification and other design specifications: (1) a page flow diagram which is one of the most essential specifications for Web applications and (2) a behavior diagram such as a UML activity diagram. We applied the proposed methods to real specifications of a Web application designed by a certain company and found several faults of the specifications that had not been detected in actual reviews.

Priority Integration for Weighted Combinatorial Testing

Priorities (weights) for parameter values can improve the effectiveness of combinatorial testing. Previous approaches have employed weights to derive high-priority test cases either earlier or more frequently. Our approach integrates these order-focused and frequency-focused prioritizations. We show that our priority integration realizes a small test suite providing high-priority test cases early and frequently in a good balance. We also propose two algorithms that apply our priority integration to existing combinatorial test generation algorithms. Experimental results using numerous test models show that our approach improves the existing approaches w.r.t. Order-focused and frequency-focused metrics, while overheads in the size and generation time of test suites are small.

Greedy combinatorial test case generation using unsatisfiable cores

Combinatorial testing aims at covering the interactions of parameters in a system under test, while some combinations may be forbidden by given constraints (forbidden tuples). In this paper, we illustrate that such forbidden tuples correspond to unsatisfiable cores, a widely understood notion in the SAT solving community. Based on this observation, we propose a technique to detect forbidden tuples lazily during a greedy test case generation, which significantly reduces the number of required SAT solving calls. We further reduce the amount of time spent in SAT solving by essentially ignoring constraints while constructing each test case, but then “amending” it to obtain a test case that satisfies the constraints, again using unsatisfiable cores. Finally, to complement a disturbance due to ignoring constraints, we implement an efficient approximative SAT checking function in the SAT solver Lingeling. Through experiments we verify that our approach significantly improves the efficiency of constraint handling in our greedy combinatorial testing algorithm.

Combinatorial Testing for Tree-Structured Test Models with Constraints

In this paper, we develop a combinatorial testing technique for tree-structured test models. First, we generalize our previous test models for combinatorial testing based on AND-XOR trees with constraints limited to a syntactic subset of propositional logic, to allow for constraints in full propositional logic. We prove that the generalized test models are strictly more expressive than the limited ones. Then we develop an algorithm for combinatorial testing for the generalized models, and show its correctness and computational complexity. We apply a tool based on our algorithm to an actual ticket gate system that is used by several large transportation companies in Japan. Experimental results show that our technique outperforms existing techniques.

Design of Prioritized N-Wise Testing

[InlineEquation not available: see fulltext.]-wise testing is a widely used technique for combinatorial interaction testing. Prioritizing testing reorders test cases by relevance, testing important aspects more thoroughly. We propose a novel technique for [InlineEquation not available: see fulltext.]-wise test case generation to satisfy the three distinct prioritization criteria of interaction coverage, weight coverage, and KL divergence. The proposed technique generates small [InlineEquation not available: see fulltext.]-wise test cases, where high-priority test cases appear early and frequently. Our early evaluation confirms that the proposed technique improves on existing techniques based on the three prioritization criteria.

Log-Based Anomaly Detection of CPS Using a Statistical Method

Detecting anomalies of a cyber physical system (CPS), which is a complex system consisting of both physical and software parts, is important because a CPS often operates autonomously in an unpredictable environment. However, because of the ever-changing nature and lack of a precise model for a CPS, detecting anomalies is still a challenging task. To address this problem, we propose applying an outlier detection method to a CPS log. By using a log obtained from an actual aquarium management system, we evaluated the effectiveness of our proposed method by analyzing outliers that it detected. By investigating the outliers with the developer of the system, we confirmed that some outliers indicate actual faults in the system. For example, our method detected failures of mutual exclusion in the control system that were unknown to the developer. Our method also detected transient losses of functionalities and unexpected reboots. On the other hand, our method did not detect anomalies that were too many and similar. In addition, our method reported rare but unproblematic concurrent combinations of operations as anomalies. Thus, our approach is effective at finding anomalies, but there is still room for improvement.

Distance-Integrated Combinatorial Testing

This paper proposes a novel approach to combinatorial test generation, which achieves an increase of not only the number of new combinations but also the distance between test cases. We applied our distance-integrated approach to a state-of-the-art greedy algorithm for traditional combinatorial test generation by using two distance metrics, Hamming distance, and a modified chi-square distance. Experimental results using numerous benchmark models show that combinatorial test suites generated by our approach using both distance metrics can improve interaction coverage for higher interaction strengths with low computational overhead.

Test Effectiveness Evaluation of Prioritized Combinatorial Testing: A Case Study

Combinatorial testing is a widely-used technique to detect system interaction failures. To improve test effectiveness with given priority weights of parameter values in a system under test, prioritized combinatorial testing constructs test suites where highly weighted parameter values appear earlier or more frequently. Such order-focused and frequency-focused combinatorial test generation algorithms have been evaluated using metrics called weight coverage and KL divergence but not sufficiently with fault detection effectiveness so far. We evaluate the fault detection effectiveness on a collection of open source utilities, applying prioritized combinatorial test generation and investigating its correlation with weight coverage and KL divergence.

Model-Based Testing of Stateful APIs with Modbat

Modbat makes testing easier by providing a user-friendly modeling language to describe the behavior of systems, from such a model, test cases are generated and executed. Modbats domain-specific language is based on Scala, its features include probabilistic and non-deterministic transitions, component models with inheritance, and exceptions. We demonstrate the versatility of Modbat by finding a confirmed defect in the currently latest version of Java, and by testing SAT solvers.

Efficiency analysis of model-based review in actual software design

In this paper, we quantitatively analyze the efficiency of the Model-Based Review (MBR) method in an actual software design from the two points of view; cost and reviewability. The MBR method is a modeling procedure for the purpose of reviewing preliminary design specifications of web-based applications. We have collected process data in applying both of the MBR method and an ordinary review to a preliminary design of a developing web-based library system. Analyzing the collected process data, we quantitatively compare the efficiency of the MBR method and that of the ordinary review. As a result of this comparative analysis, we show that the MBR method is superior to the ordinary review in terms of not only reviewability but also cost through the experimental design process.