Rawad Abou Assi

Cleansing Test Suites from Coincidental Correctness to Enhance Fault-Localization

Researchers have argued that for failure to be observed the following three conditions must be met: 1) the defect is executed, 2) the program has transitioned into an infectious state, and 3) the infection has propagated to the output. Coincidental correctness arises when the program produces the correct output, while conditions 1) and 2) are met but not 3). In previous work, we showed that coincidental correctness is prevalent and demonstrated that it is a safety reducing factor for coverage-based fault localization. This work aims at cleansing test suites from coincidental correctness to enhance fault localization. Specifically, given a test suite in which each test has been classified as failing or passing, we present three variations of a technique that identify the subset of passing tests that are likely to be coincidentally correct. We evaluated the effectiveness of our techniques by empirically quantifying the following: 1) how accurately did they identify the coincidentally correct tests, 2) how much did they improve the effectiveness of coverage-based fault localization, and 3) how much did coverage decrease as a result of applying them. Using our better performing technique and configuration, the safety and precision of fault-localization was improved for 88% and 61% of the programs, respectively.

Prevalence of coincidental correctness and mitigation of its impact on fault localization

Researchers have argued that for failure to be observed the following three conditions must be met: <i>C</i>_<i>R</i> = the defect was reached; <i>C</i>_<i>I</i> = the program has transitioned into an infectious state; and <i>C</i>_<i>P</i> = the infection has propagated to the output. <i>Coincidental Correctness</i> (CC) arises when the program produces the correct output while condition C_R is met but not C_P. We recognize two forms of coincidental correctness, weak and strong. In <i>weak CC</i>, C_R is met, whereas C_I might or might not be met, whereas in <i>strong</i> <i>CC</i>, both C_R and C_I are met. In this work we first show that CC is prevalent in both of its forms and demonstrate that it is a safety reducing factor for <i>Coverage-Based Fault Localization</i> (CBFL). We then propose two techniques for cleansing test suites from coincidental correctness to enhance CBFL, given that the test cases have already been classified as failing or passing. We evaluated the effectiveness of our techniques by empirically quantifying their accuracy in identifying weak CC tests. The results were promising, for example, the better performing technique, using 105 test suites and statement coverage, exhibited 9% false negatives, 30% false positives, and no false negatives nor false positives in 14.3% of the test suites. Also using 73 test suites and more complex coverage, the numbers were 12%, 19%, and 15%, respectively.

Identifying Failure-Correlated Dependence Chains

Coverage-based fault localization techniques aim at locating faulty code by first identifying the executing program elements that correlate the most with failure. It often happens that the correlation measure of such elements is not high enough to successfully guide the developer to the fault. This shortcoming is likely due to the fact that the covered program elements are simple (e.g., statements, branches, or def-use pairs), and thus, cannot characterize most defects that are typically non trivial. In this work we aim at identifying short dependence chains that are highly correlated with failure, which we term failure correlated dependence chains. For more effective fault localization, both the number and length of these failure correlated chains should be minimal. In addition to considering data and control dependences, we augment each chain by computing a set of predicates involving the source values and target values of its edges. This supplementary state information can potentially help identify failure-correlated chains that are shorter in length and can better assist in locating the faulty code. We used 18 versions of the Siemens test suite to evaluate the effectiveness of our technique in comparison to when statement coverage is used. In our experimental study we empirically quantified the following: How often is our technique successful at identifying chains that are relatively more correlated with failure? Our results were promising as the technique successfully identified more correlated chains in 17 out of 18 versions.

Enhancing Fault Localization via Multivariate Visualization

The majority of dynamic software analyses are implemented in the form of fully-automated techniques. Given the limited success of many of these techniques, we explore the use of visualization as the basis for alternative techniques. Specifically, we investigate the use of multivariate visualization scatter plots, which aim at presenting high dimensional data in low dimensions (e.g., 2D). For example, to visualize the similarity between test cases, where a test is represented as a scatter point, the execution profiles induced by the test cases are compared in order to calculate similarity metrics that will characterize the distances between the scatter points. This type of scatter plots was previously presented by other researchers who also suggested their use in several software analyses. This work considers these scatter plots in: 1) identifying coincidentally correct tests which are a safety reducing factor in coverage based fault localization, and 2) outlining a user aided visualization-based fault localization technique. Other applications of multivariate visualization to software analysis will also be discussed.

Generating profile-based signatures for online intrusion and failure detection

Context: Program execution profiles have been extensively and successfully used in several dynamic analysis fields such as software testing and fault localization. Objective: This paper presents a pattern-matching approach implemented as an application-based intrusion (and failure) detection system that operates on signatures generated from execution profiles. Such signatures are not descriptions of exploits, i.e. they do not depend on the syntax or semantics of the exploits, but instead are descriptions of program events that correlate with the exploitation of program vulnerabilities. Method: A vulnerability exploit is generally correlated with the execution of a combination of program elements, such as statements, branches, and definition-use pairs. In this work we first analyze the execution profiles of a vulnerable application in order to identify such suspicious combinations, define signatures that describe them, and consequently deploy these signatures within an intrusion detection system that performs online signature matching. Results: To evaluate our approach, which is also applicable to online failure detection, we implemented it for the Java platform and applied it onto seven open-source applications containing 30 vulnerabilities/defects for the purpose of the online detection of attacks/ failures. Our results showed that our approach worked very well for 26 vulnerabilities/defects (86.67%) and the overhead imposed by the system is somewhat acceptable as it varied from 46% to 102%. The exhibited average rates of false negatives and false positives were 0.43% and 1.03%, respectively. Conclusion: Using profile-based signatures for online intrusion and failure detection was shown to be effective.

State Profiling of Internal Variables

Execution profiles form the basis of many dynamic program analysis techniques developed to solve problems in fields such as software testing, fault localization, and program comprehension. A typical profile comprises information that approximates the execution path of a program, specifically, the frequency of occurrence of program elements that are structural in nature, such as statements, branches, and def-use pairs. It is plausible that some defects might not be characterized by a failing path or a set of structural program elements. In such cases, execution profiles that are based on structural elements would not be of much benefit. We argue for the need to leverage the state of the subject program, i.e., to analyze the values of representative internal variables that approximate the program state. For that purpose we: 1) built a profiling tool that captures the values taken by select variables, 2) devised three metrics to quantify the dissimilarities between our state-based execution profiles, and 3) conducted ten case studies to demonstrate the effectiveness of our tool and metrics. The majority of our case studies (8 out of 10), using all metrics, showed that failing profiles are most dissimilar from passing profiles.

Does Principal Component Analysis Improve Cluster-Based Analysis?

Researchers in the dynamic program analysis field have extensively used cluster analysis to address various problems. Typically, the clustering techniques are applied onto execution profiles having high dimensionality (i.e., involving a large number of profiling elements), sometimes in the order of thousands or even hundreds of thousands. Our concern is that the high number of profiling elements might diminish the effectiveness of the clustering process, which led us to explore the use of dimensionality reduction techniques as a preprocessing step to clustering. Specifically, in this work, we used PCA (Principal Component Analysis) as a dimensionality reduction technique and investigated its impact on two cluster-based analysis techniques, one aiming at identifying coincidentally correct tests, and the other at test suite minimization. In other words, we tried to assess whether PCA improves cluster-based analysis. Our experimental results showed that the impact was positive on the first technique, but inconclusive on the second, which calls for further investigation in the future.

UCov: a user‐defined coverage criterion for test case intent verification

The goal of regression testing is to ensure that the behaviour of existing code, believed correct by previous testing, is not altered by new program changes. This paper argues that the primary focus of regression testing should be on code associated with (1) earlier bug fixes and (2) particular application scenarios considered to be important by the developer or tester. Existing coverage criteria do not enable such focus, for example, 100% branch coverage does not guarantee that a given bug fix is exercised or a given application scenario is tested. Therefore, there is a need for a new and complementary coverage criterion in which the user can definea test requirement characterizing a given behaviour to be covered as opposed to choosing from a pool of pre‐defined and generic program elements. This paper proposes this new methodology and calls it UCov, a user‐defined coverage criterion wherein a test requirement is an execution pattern of program elements, and possibly predicates, that a test case must satisfy. The proposed criterion is not meant to replace existing criteria, but to complement them as it focuses the testing on important code patterns that could go untested otherwise. UCov supports test case intent verification. For example, following a bug fix, the testing team may augment the regression suite with the test case that revealed the bug. However, this test case might become obsolete due to code modifications not related to the bug. But if a test requirement characterizing the bug was defined by the user, UCov would determine that test case intent verification failed. The UCov methodology was implemented for the Java platform, was successfully applied onto 10 real‐life case studies and was shown to have advantages over JUnit. The implementation comprises the following tools: (1) TRSpec: allows the user to easily specify complex test requirements; (2) TRCheck: checks whether user‐defined test requirements were satisfied, that is, supports test case intent verification; and (3) TRMigrate: migrates user‐defined test requirements to subsequent versions of a given program. Copyright

Reducing execution profiles: techniques and benefits

The interest in leveraging data mining and statistical techniques to enable dynamic program analysis has increased tremendously in recent years. Researchers have presented numerous techniques that mine and analyze execution profiles to assist software testing and other reliability enhancing approaches. Previous empirical studies have shown that the effectiveness of such techniques is likely to be impacted by the type of profiled program elements. This work further studies the impact of the characteristics of execution profiles by focusing on their size; noting that a typical profile comprises a large number of program elements, in the order of thousands or higher. Specifically, the authors devised six reduction techniques and comparatively evaluated them by measuring the following: (1) reduction rate; (2) information loss; (3) impact on two applications of dynamic program analysis, namely, cluster‐based test suite minimization (App‐I), and profile‐based online failure and intrusion detection (App‐II). The results were promising as the following: (a) the average reduction rate ranged from 92% to 98%; (b) three techniques were lossless and three were slightly lossy; (c) reducing execution profiles exhibited a major positive impact on the effectiveness and efficiency of App‐I; and (d) reduction exhibited a positive impact on the efficiency of App‐II, but a minor negative impact on its effectiveness. Copyright

Lossless Reduction of Execution Profiles Using a Genetic Algorithm

Typically, an execution profile comprises a large number of profiling elements, in the order of thousands or more, amongst which a considerable proportion are redundant. One factor behind this redundancy is likely to be the transitivity relationships induced by control and data dependences. Reducing such redundancy is desirable for several reasons. In this work we propose a reduction mechanism based on a genetic algorithm to eliminate redundancy in execution profiles. Such mechanism is lossless in the sense that the original execution profiles could be entirely inferred from the reduced ones. We evaluated our approach empirically by measuring its impact on two types of analyses that leverage execution profiles, namely clustering and greedy test suite minimization. The experiments we conducted involved 8 subject programs from the SIR repository, each seeded with a number of faults. The results were very promising as the reduction rate ranged from 94% to 99% with a negligible deterioration in the quality of clustering and minimization.