Donghwan Shin

Automated test case generation for FBD programs implementing reactor protection system software

Automated and effective testing for function block diagram (FBD) programs has become an important issue, as FBD is increasingly used in implementing safety‐critical systems. This work describes an automated test case generation technique for FBD programs and its associated tool—FBDTester. Given an FBD program and desired test coverage criteria, FBDTester generates test requirements and invokes the Satisfiability Modulo Theories solver iteratively to derive a set of test cases. An industrial case study using reactor protection system software shows that the automatically generated test suites detected at least 82% of the known faults, whereas manually generated test cases only detected approximately 35%. Mutation analysis revealed that the automatically generated test suites substantially outperformed manually generated ones. Although test sequence generation requires some manual effort in the current FBDTester, it is apparent that the proposed approach significantly improves the efficiency and the reliability of FBD testing. Copyright

Empirical evaluation on FBD model-based test coverage criteria using mutation analysis

Function Block Diagram (FBD), one of the PLC programming languages, is a graphical modeling language which has been increasingly used to implement safety-critical software such as nuclear reactor protection software. With increased importance of structural testing for FBD models, FBD model-based test coverage criteria have been introduced. In this paper, we empirically evaluate the fault detection effectiveness of the FBD coverage criteria using mutation analysis. We produce 1800 test suites satisfying the FBD criteria and generate more than 600 mutants automatically for the target industrial FBD models. Then we evaluate mutant detection of the test suites to assess the fault detection effectiveness of the coverage criteria. Based on the experimental results, we analyze strengths and weaknesses of the FBD coverage criteria, and suggest possible improvements for the test coverage criteria.

Human Resource Allocation in Software Project with Practical Considerations

Software planning is very important for the success of a software project. Even if the same developers work on the same project, the time span of the project and the quality of software may change ...

A Theoretical Framework for Understanding Mutation-Based Testing Methods

In the field of mutation analysis, mutation is the systematic generation of mutated programs (i.e., mutants) from an original program. The concept of mutation has been widely applied to various testing problems, including test set selection, fault localization, and program repair. However, surprisingly little focus has been given to the theoretical foundation of mutation-based testing methods, making it difficult to understand, organize, and describe various mutation-based testing methods. This paper aims to consider a theoretical framework for understanding mutation-based testing methods. While there is a solid testing framework for general testing, this is incongruent with mutation-based testing methods, because it focuses on the correctness of a program for a test, while the essence of mutation-based testing concerns the differences between programs (including mutants) for a test. In this paper, we begin the construction of our framework by defining a novel testing factor, called a test differentiator, to transform the paradigm of testing from the notion of correctness to the notion of difference. We formally define behavioral differences of programs for a set of tests as a mathematical vector, called a d-vector. We explore the multi-dimensional space represented by d-vectors, and provide a graphical model for describing the space. Based on our framework and formalization, we interpret existing mutation-based fault localization methods and mutant set minimization as applications, and identify novel implications for future work.

Diversity-Aware Mutation Adequacy Criterion for Improving Fault Detection Capability

Many existing testing techniques adopt diversity as an important criterion for the selection and prioritization of tests. However, mutation adequacy has been content with simply maximizing the number of mutants that have been killed. We propose a novel mutation adequacy criterion that considers the diversity in the relationship between tests and mutants, as well as whether mutants are killed. Intuitively, the proposed criterion is based on the notion that mutants can be distinguished by the sets of tests that kill them. A test suite is deemed adequate by our criterion if the test suite distinguishes all mutants in terms of their kill patterns. Our hypothesis is that, simply by using a stronger adequacy criterion, it is possible to improve fault detection capabilities of mutation-adequate test suites. The empirical evaluation selects tests for real world applications using the proposed mutation adequacy criterion to test our hypothesis. The results show that, for real world faults, test suites adequate to our criterion can increase the fault detection success rate by up to 76.8 percentage points compared to test suites adequate to the traditional criterion.

Quality Based Software Project Staffing and Scheduling with Cost Bound

Software project planning is becoming more complicated and important as the size of software project grows. Many approaches have been proposed to help project managers by providing optimal staffing and scheduling in terms of minimizing the cost (i.e., necessary expanse) or time (i.e., time span or duration) required for the software project. Unfortunately, the software quality, another critical factor in software project planning, is largely overlooked in previous work. In this paper, we propose the quality based software project staffing and scheduling approach using a genetic algorithm (GA). We define a quality score by considering practical issues in software project planning in addition to task severity and defect amplification model. Further, the cost is utilized as a cost-bound in the GA to consider not only quality but also cost. Case study shows that the proposed approach improves the quality while the cost is optimized as the same as the cost-based approach. In other words, we provide better software project plans considering both cost and quality for software project managers. Also, we show the relationship between the quality and the cost in terms of software project planning.

Are mutation scores correlated with real fault detection?: a large scale empirical study on the relationship between mutants and real faults

Empirical validation of software testing studies is increasingly relying on mutants. This practice is motivated by the strong correlation between mutant scores and real fault detection that is reported in the literature. In contrast, our study shows that correlations are the results of the confounding effects of the test suite size. In particular, we investigate the relation between two independent variables, mutation score and test suite size, with one dependent variable the detection of (real) faults. We use two data sets, CoreBench and De-fects4J, with large C and Java programs and real faults and provide evidence that all correlations between mutation scores and real fault detection are weak when controlling for test suite size. We also found that both independent variables significantly influence the dependent one, with significantly better fits, but overall with relative low prediction power. By measuring the fault detection capability of the top ranked, according to mutation score, test suites (opposed to randomly selected test suites of the same size), we found that achieving higher mutation scores improves significantly the fault detection. Taken together, our data suggest that mutants provide good guidance for improving the fault detection of test suites, but their correlation with fault detection are weak.

Mutation analysis for system of systems policy testing

A System of Systems (SoS) is a set of the constituent systems (CS) which has managerial and operational independence. To address an SoS-level goal that cannot be satisfied by each CS, an SoS policy guides or forces the CSs to collaborate with each other. If there is a fault in the SoS policy, SoS may fail to reach its goal, even if there is no fault in the CSs. Such a call for SoS policy testing leads to an essential question---how can testers evaluate the effectiveness of test cases? In this paper, we suggest a mutation analysis approach for SoS policy testing. Mutation analysis is a systematic way of evaluating test cases using artificial faults called mutants. As a general mutation framework for SoS policy testing, we present an overview of mutation analysis in SoS policy testing as well as the key aspects that must be defined in practice. To demonstrate the applicability of the proposed approach, we provide a case study using a traffic management SoS with the Simulation of Urban Mobility (SUMO) simulator. The results show that the mutation analysis is effective at evaluating fault detection effectiveness of test cases for SoS policies at a reasonable cost.

A runtime verification framework for dynamically adaptive multi-agent systems

Dynamically adaptive multi-agent systems (DAMS) consist of multiple agents that adapt to changing system and environmental conditions in order to achieve collaborative goals. As DAMS are found in applications across various domains, ensuring the correct and safe adaptations of DAMS has become more important. Formal verification techniques such as model checking present a promising approach to guaranteeing the correctness of a software system with respect to certain system requirements. Previous works on formal verification for dynamically adaptive system or multi-agent system, however, have not addressed the runtime and collaborative nature inherent to DAMS operations. This work proposes a runtime verification framework for DAMS (DAMS-RV) based on an adaptive feedback loop, which is activated for each adaptation that system makes after a change in the system or environment. The proposed framework is described using a collaborative nurse agent system as a running example. A case study with an application scenario provides insights into how DAMS-RV can serve as a feasible and effective framework for DAMS verification.

Modeling and verification for different types of system of systems using PRISM

A System of Systems (SoS) consists of complex Constituent Systems (CSs), which have managerial and operational independence. As the CSs have independence, the degree of authority to control the CSs varies depending on the different types of SoS, such as directed, acknowledged, and collaborative SoS. While many researchers have attempted to model SoSs and verify SoS-level goals, little attention has been paid to how different types of SoSs are characterized and realized during modeling and verification. Using a mass casualty incident response system as a common working example for different types of a SoS, we model the types of SoS with probabilistic models of PRISM, which can capture uncertain behaviors of an SoS in a quantitative manner and perform statistical model checking in order to verify to which extent each type of SoS achieves the SoS-level goal. Differences between the different types of SoS are demonstrated in the probabilistic models and analyzed by statistical model checking. This modeling and verification can help SoS-level managers to identify the appropriate SoS types to achieve the SoS-level goal.