Moonzoo Kim

Java-MaC: A Run-Time Assurance Approach for Java Programs

We describe Java-MaC, a prototype implementation of the Monitoring and Checking (MaC) architecture for Java programs. The MaC architecture provides assurance that the target program is running correctly with respect to a formal requirements specification by monitoring and checking the execution of the target program at run-time. MaC bridges the gap between formal verification, which ensures the correctness of a design rather than an implementation, and testing, which does not provide formal guarantees about the correctness of the system.Use of formal requirement specifications in run-time monitoring and checking is the salient aspect of the MaC architecture. MaC is a lightweight formal method solution which works as a viable complement to the current heavyweight formal methods. In addition, analysis processes of the architecture including instrumentation of the target program, monitoring, and checking are performed fully automatically without human direction, which increases the accuracy of the analysis. Another important feature of the architecture is the clear separation between monitoring implementation-dependent low-level behaviors and checking high-level behaviors, which allows the reuse of a high-level requirement specification even when the target program implementation changes. Furthermore, this separation makes the architecture modular and allows the flexibility of incorporating third party tools into the architecture. The paper presents an overview of the MaC architecture and a prototype implementation Java-MaC.

Directed test suite augmentation: techniques and tradeoffs

Test suite augmentation techniques are used in regression testing to identify code elements affected by changes and to generate test cases to cover those elements. Our preliminary work suggests that several factors influence the cost and effectiveness of test suite augmentation techniques. These include the order in which affected elements are considered while generating test cases, the manner in which existing regression test cases and newly generated test cases are used, and the algorithm used to generate test cases. In this work, we present the results of an empirical study examining these factors, considering two test case generation algorithms (concolic and genetic). The results of our experiment show that the primary factor affecting augmentation is the test case generation algorithm utilized; this affects both cost and effectiveness. The manner in which existing and newly generated test cases are utilized also has a substantial effect on efficiency but a lesser effect on effectiveness. The order in which affected elements are considered turns out to have relatively few effects when using concolic test case generation, but more substantial effects when using genetic test case generation.

Feature-oriented re-engineering of legacy systems into product line assets: a case study

Home service robots have a wide range of potential applications, such as home security, patient caring, cleaning, etc. The services provided by the robots in each application area are being defined as markets are formed and, therefore, they change constantly. Thus, robot applications need to evolve both quickly and flexibly adopting frequently changing requirements. This makes software product line framework ideal for the domain of home service robots. Unfortunately, however, robot manufacturers often focus on developing technical components (e.g., vision recognizer and speech processor) and then attempt to develop robots by integrating these components in an ad-hoc way. This practice produces robot applications that are hard to re-use and evolve when requirements change. We believe that re-engineering legacy robot applications into product line assets can significantly enhance reusability and evolvability. In this paper, we present our experience of re-engineering legacy home service robot applications into product line assets through feature modeling and analysis. First, through reverse engineering, we recovered architectures and components of the legacy applications. Second, based on the recovered information and domain knowledge, we reconstructed a feature model for the legacy applications. Anticipating changes in business opportunities or technologies, we restructured and refined the feature model to produce a feature model for the product line. Finally, based on the refined feature model and engineering principles we adopted for asset development, we designed a new architecture and components for robot applications.

Ask the Mutants: Mutating Faulty Programs for Fault Localization

We present MUSE (MUtation-baSEd fault localization technique), a new fault localization technique based on mutation analysis. A key idea of MUSE is to identify a faulty statement by utilizing different characteristics of two groups of mutants-one that mutates a faulty statement and the other that mutates a correct statement. We also propose a new evaluation metric for fault localization techniques based on information theory, called Locality Information Loss (LIL): it can measure the aptitude of a localization technique for automated fault repair systems as well as human debuggers. The empirical evaluation using 14 faulty versions of the five real-world programs shows that MUSE localizes a fault after reviewing 7.4 statements on average, which is about 25 times more precise than the state-of-the-art SBFL technique Op2.

Testing concurrent programs to achieve high synchronization coverage

The effectiveness of software testing is often assessed by measuring coverage of some aspect of the software, such as its code. There is much research aimed at increasing code coverage of sequential software. However, there has been little research on increasing coverage for concurrent software. This paper presents a new technique that aims to achieve high coverage of concurrent programs by generating thread schedules to cover uncovered coverage requirements. Our technique first estimates synchronization-pair coverage requirements, and then generates thread schedules that are likely to cover uncovered coverage requirements. This paper also presents a description of a prototype tool that we implemented in Java, and the results of a set of studies we performed using the tool on a several open-source programs. The results show that, for our subject programs, our technique achieves higher coverage faster than random testing techniques; the estimation-based heuristic contributes substantially to the effectiveness of our technique.

A Scalable Distributed Concolic Testing Approach: An Empirical Evaluation

Although testing is a standard method for improving the quality of software, conventional testing methods often fail to detect faults. Concolic testing attempts to remedy this by automatically generating test cases to explore execution paths in a program under test, helping testers achieve greater coverage of program behavior in a more automated fashion. Concolic testing, however, consumes a significant amount of computing time to explore execution paths, which is an obstacle toward its practical application. To address this limitation, we have developed a scalable distributed concolic testing framework that utilizes large numbers of computing nodes to generate test cases in a scalable manner. In this paper, we present the results of an empirical study that shows that the proposed framework can achieve a several orders-of-magnitude increase in test case generation speed compared to the original concolic approach, and also demonstrates clear potential for scalability.

A Comparative Study of Software Model Checkers as Unit Testing Tools: An Industrial Case Study

Conventional testing methods often fail to detect hidden flaws in complex embedded software such as device drivers or file systems. This deficiency incurs significant development and support/maintenance cost for the manufacturers. Model checking techniques have been proposed to compensate for the weaknesses of conventional testing methods through exhaustive analyses. Whereas conventional model checkers require manual effort to create an abstract target model, modern software model checkers remove this overhead by directly analyzing a target C program, and can be utilized as unit testing tools. However, since software model checkers are not fully mature yet, they have limitations according to the underlying technologies and tool implementations, potentially critical issues when applied in industrial projects. This paper reports our experience in applying Blast and CBMC to testing the components of a storage platform software for flash memory. Through this project, we analyzed the strong and weak points of two different software model checking technologies in the viewpoint of real-world industrial application-counterexample-guided abstraction refinement with predicate abstraction and SAT-based bounded analysis.

Detecting Concurrency Errors in Client-Side Java Script Web Applications

As web technologies have evolved, the complexity of dynamic web applications has increased significantly and web applications suffer concurrency errors due to unexpected orders of interactions among web browsers, users, the network, and so forth. In this paper, we present WAVE (Web Applications Virtual Environment), a testing framework to detect concurrency errors in client-side web applications written in JavaScript. WAVE generates various sequences of operations as test cases for a web application and executes a sequence of operations by dynamically controlling interactions of a target web application with the execution environment. We demonstrate that WAVE is effective and efficient for detecting concurrency errors through experiments on eight examples and five non-trivial real-world web applications.

Understanding user understanding: determining correctness of generated program invariants

Recently, work has begun on automating the generation of test oracles, which are necessary to fully automate the testing process. One approach to such automation involves dynamic invariant generation which extracts invariants from program executions. To use such invariants as test oracles, however, it is necessary to distinguish correct from incorrect invariants, a process that currently requires human intervention. In this work we examine this process. In particular, we examine the ability of 30 users, across two empirical studies, to classify invariants generated from three Java programs. Our results indicate that users struggle to classify generated invariants: on average, they misclassify 9.1% to 31.7% of correct invariants and 26.1%-58.6% of incorrect invariants. These results contradict prior studies that suggest that classification by users is easy, and indicate that further work needs to be done to bridge the gap between the effectiveness of dynamic invariant generation in theory, and the ability of users to apply it in practice. Along these lines, we suggest several areas for future work.

Formal Verification of Robot Movements - a Case Study on Home Service Robot SHR100

Home service robots have received much attention from both academia and industry because home service robots have wide range of potential applications such as home security, cleaning, etc. The robots need to add or update services frequently according to the changing needs of human users. Furthermore, reactive nature of the robots add complexity to develop robot applications. These challenges raise safety issues seriously. Considering that safe operation of home service robots is crucial, current practice of validating robot applications is, however, not mature enough for wide deployment of home service robots. In this paper, we present our experience of developing and formally verifying discrete control software of Samsung Home Robot (SHR) using Esterel. We give a brief background on Esterel, then illuminate our result in formally verifying stopping behavior of SHR. Through the verification, we could detect and solve a feature interaction problem which caused the robot not to stop when a user commanded the robot to stop.