Ruizhi Gao

A Survey on Software Fault Localization

Software fault localization, the act of identifying the locations of faults in a program, is widely recognized to be one of the most tedious, time consuming, and expensive - yet equally critical - activities in program debugging. Due to the increasing scale and complexity of software today, manually locating faults when failures occur is rapidly becoming infeasible, and consequently, there is a strong demand for techniques that can guide software developers to the locations of faults in a program with minimal human intervention. This demand in turn has fueled the proposal and development of a broad spectrum of fault localization techniques, each of which aims to streamline the fault localization process and make it more effective by attacking the problem in a unique way. In this article, we catalog and provide a comprehensive overview of such techniques and discuss key issues and concerns that are pertinent to software fault localization as a whole.

The DStar Method for Effective Software Fault Localization

Effective debugging is crucial to producing reliable software. Manual debugging is becoming prohibitively expensive, especially due to the growing size and complexity of programs. Given that fault localization is one of the most expensive activities in program debugging, there has been a great demand for fault localization techniques that can help guide programmers to the locations of faults. In this paper, a technique named DStar (D*) is proposed which can suggest suspicious locations for fault localization automatically without requiring any prior information on program structure or semantics. D* is evaluated across 24 programs, and is compared to 38 different fault localization techniques. Both single-fault and multi-fault programs are used. Results indicate that D* is more effective at locating faults than all the other techniques it is compared to. An empirical evaluation is also conducted to illustrate how the effectiveness of D* increases as the exponent * grows, and then levels off when the exponent * exceeds a critical value. Discussions are presented to support such observations.

Software Fault Localization Using DStar (D

Effective debugging is crucial to producing dependable software. Manual debugging is becoming prohibitively expensive, especially due to the growing size and complexity of programs. Given that fault localization is one of the most expensive activities in program debugging, there has been a great demand for fault localization techniques that can help guide programmers to the locations of faults. In this paper a technique named DStar (D*), which has its origins rooted in similarity coefficient-based analysis, is proposed, which can identify suspicious locations for fault localization automatically without requiring any prior information on program structure or semantics. D* is evaluated across 21 programs and is compared to 16 different fault localization techniques. Both single-fault and multi-fault programs are used. Results indicate that D* is more effective at locating faults than all the other techniques it is compared to.

Genetic Algorithm-based Test Generation for Software Product Line with the Integration of Fault Localization Techniques

In response to the highly competitive market and the pressure to cost-effectively release good-quality software, companies have adopted the concept of software product line to reduce development cost. However, testing and debugging of each product, even from the same family, is still done independently. This can be very expensive. To solve this problem, we need to explore how test cases generated for one product can be used for another product. We propose a genetic algorithm-based framework which integrates software fault localization techniques and focuses on reusing test specifications and input values whenever feasible. Case studies using four software product lines and eight fault localization techniques were conducted to demonstrate the effectiveness of our framework. Discussions on factors that may affect the effectiveness of the proposed framework is also presented. Our results indicate that test cases generated in such a way can be easily reused (with appropriate conversion) between different products of the same family and help reduce the overall testing and debugging cost.

Effective software fault localization using predicted execution results

Software has become ubiquitous in our daily lives, and with its increasing functionality and complexity comes a frequently tedious and prolonged debugging process. Of the three activities in program debugging (failure detection, fault localization, and bug fixing), the focus of this paper is on the first, failure detection, under the condition that there is no test oracle that can be used to automatically determine the success or failure of all the executions. More precisely, the outputs for many executions have to be verified manually, or the expected outputs are not even available. We want to determine whether there is a solution to help programmers predict the execution results. How good are these predicted results when they are used to help programmers find the locations of bugs? A framework is proposed to reduce the effort on output verification using a strategy based on the Hamming distance or K-Means clustering to predict results of test executions. Such data and the statement coverage of each test case are used to compute the suspiciousness of each statement according to a fault localization technique and produce a ranking for examination to locate bugs. Case studies using 22 programs and seven fault localization techniques were conducted to evaluate the fault localization effectiveness of the proposed framework on 1203 faulty versions, some of which have a single bug and others with multiple bugs. A discussion on factors that may affect the accuracy of execution result prediction and the resulting fault localization effectiveness is also presented. Our data suggests that, in general, with respect to fault localization techniques using execution results verified against the expected outputs, those using predicted execution results can be even more effective than (by examining a smaller number of statements to locate the first faulty statement) or as good as the former (the verified).

An empirical study on constraint optimization techniques for test generation

Constraint solving is a frequent, but expensive operation with symbolic execution to generate tests for a program. To improve the efficiency of test generation using constraint solving, four optimization techniques are usually applied to existing constraint solvers, which are constraint independence, constraint set simplification, constraint caching, and expression rewriting. In this paper, we conducted an empirical study, using these four constraint optimization techniques in a well known test generation tool KLEE with 77 GNU Coreutils applications, to systematically investigate how these optimization techniques affect the efficiency of test generation. The experimental results show that these constraint optimization techniques as well as their combinations cannot improve the efficiency of test generation significantly for ALL-SIZED programs. Moreover, we studied the constraint optimization techniques with respect to two static metrics, lines of code (LOC) and cyclomatic complexity (CC), of programs. The experimental results show that the “constraint set simplification” technique can improve the efficiency of test generation significantly for the programs with high LOC and CC values. The “constraint caching” optimization technique can improve the efficiency of test generation significantly for the programs with low LOC and CC values. Finally, we propose four hybrid optimization strategies and practical guidelines based on different static metrics.创新点我们使用KLEE执行了77个GNU Coreutils程序,用于系统的研究4种流行的约束优化技术如何影响测试用例生成的效率。结果表明,对于所有程序,这些约束优化技术及它们的组合不能大幅提高测试用例生成的效率。 此外,我们研究了程序的两个静态指标(代码行和圈复杂度)跟约束优化技术的关系。结果表明,对于大规模及高圈复杂度程序,使用“约束集简化”技术,对于小规模及低圈复杂度程序,使用“约束缓存”技术,都可以显著提高测试生成效率。最后,基于不同的静态指标,我们提出了四种混合优化策略和指导方针。

Applying Combinatorial Testing in Industrial Settings

Combinatorial testing (CT) is a black-box-based technique to generate a small number of test cases with a focus on covering various interactions among input parameters of a software system. How effective is CT in practice? What are the challenges and issues presented by the process of applying CT? How can these challenges and issues be overcome to improve the application of CT? Although CT has attracted attention from both academia and industry, these questions have not been addressed or researched in depth based on an empirical study in industrial settings. From January 2016 to February 2016, we have worked with CEPREI, an authoritative software testing company in China, on testing three real-life software systems using CT. Throughout the process, we have generated 601 test cases and found 33 bugs in total. The results demonstrate that CT not only detects more bugs but also requires less time in designing test cases for a specific system comparing to traditional function coverage-based testing. This paper will present our empirical study thoroughly and give an insightful analysis to investigate the advantages as well as the possible challenges of applying CT.

An empirical study of requirements-based test generation on an automobile control system

Requirements-based test generation techniques have been widely used in industry to help practitioners generate appropriate test cases to ensure that their software systems behave according to customer expectations. These tests provide a solid foundation for functional testing -- a critical, essential step that must be performed properly for quality assurance of any software system. In this paper, we present a study, in collaboration with Hyundai Motor Company, on a real-life industrial software system used to control the drivers demand torque of an automobile. Since any hazardous incident caused by this mechanism may result in significant property loss or even fatalities, its safe and reliable operation becomes absolutely imperative. To overcome this problem, we propose a framework including different requirements-based test generation techniques such as equivalence class partitioning (ECP), boundary value analysis (BVA), a choice relation framework, and predicate testing-based BOR, BRO, and BRE strategies. In addition, a tool named C-Set has been developed to support the BOR, BRO and BRE-based automatic test generation. Results from our study indicate that weaknesses in any of the above techniques can be mitigated by exploiting the strengths of the other techniques. As a result, a set of high quality test cases for the software being studied can be efficiently generated to help Hyundai engineers improve their productivity.

WAS: A weighted attribute-based strategy for cluster test selection

In past decades, many techniques have been proposed to generate and execute test cases automatically. However, when a test oracle does not exist, execution results have to be examined manually. With increasing functionality and complexity of todays software, this process can be extremely time-consuming and mistake-prone. A CTS-based (cluster test selection) strategy provides a feasible solution to mitigate such deficiency by examining the execution results only with respect to a small number of selected test cases. It groups test cases with similar execution profiles into the same cluster and selects them from each cluster. Some well-known CTS-based strategies are one per cluster, n (a predefined value which is greater than 1) per cluster, adaptive sampling, and execution-spectra-based sampling (ESBS). The ultimate goal is to reduce testing cost by quickly identifying the executions that are likely to fail. However, improperly grouping the test cases will significantly diminish the effectiveness of these strategies (by examining results of more successful executions and fewer failed executions). To overcome this problem, we propose a weighted attribute-based strategy (WAS). Instead of clustering test cases based on the similarity of their execution profiles only once like the aforementioned CTS-based strategies, WAS will conduct more than one iteration of clustering using weighted execution profiles by also considering the suspiciousness of each program element (statement, basic block, decision, etc.), where the suspiciousness in terms of the likelihood of containing bugs can be computed by using various software fault localization techniques. Case studies using seven programs (make, ant, sed, flex, grep, gzip, and space) and four CTS-based strategies (one per cluster sampling, n per cluster sampling, adaptive sampling, and ESBS) were conducted to evaluate the effectiveness of WAS on 184 faulty versions containing either single or multiple bugs. Experimental results suggest that the proposed WAS strategy outperforms other four CTS-based strategies with respect to both recall and precision such that output verification is focused more strongly on failed executions.

Software Safety Standards: Evolution and Lessons Learned

As safety issues occur in many domains, software safety standards provide guidelines for development of software systems that operate in safety-critical environments. However, evolution of existing software safety standards diverges under various circumstances and environments. To understand the purpose of these standards on their domains and the effect of changing the environment on evolution of these standards, we conducted a survey on the history of the families of DO-178 (Commercial avionics), MIL-STD-882 (US Department of Defense), and DEF-STAN 00-56 (UK Ministry of Defense). Additionally, we learned that even in different environments, there are certain features in common that are preferred by industry and would likely be added to newer versions of the standard. In other words, these features are very likely to be must-haves when constructing new standards in the future.