Akbar Siami Namin

Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria

The empirical assessment of test techniques plays an important role in software testing research. One common practice is to seed faults in subject software, either manually or by using a program that generates all possible mutants based on a set of mutation operators. The latter allows the systematic, repeatable seeding of large numbers of faults, thus facilitating the statistical analysis of fault detection effectiveness of test suites; however, we do not know whether empirical results obtained this way lead to valid, representative conclusions. Focusing on four common control and data flow criteria (block, decision, C-use, and P-use), this paper investigates this important issue based on a middle size industrial program with a comprehensive pool of test cases and known faults. Based on the data available thus far, the results are very consistent across the investigated criteria as they show that the use of mutation operators is yielding trustworthy results: generated mutants can be used to predict the detection effectiveness of real faults. Applying such a mutation analysis, we then investigate the relative cost and effectiveness of the above-mentioned criteria by revisiting fundamental questions regarding the relationships between fault detection, test suite size, and control/data flow coverage. Although such questions have been partially investigated in previous studies, we can use a large number of mutants, which helps decrease the impact of random variation in our analysis and allows us to use a different analysis approach. Our results are then; compared with published studies, plausible reasons for the differences are provided, and the research leads us to suggest a way to tune the mutation analysis process to possible differences in fault detection probabilities in a specific environment

The influence of size and coverage on test suite effectiveness

We study the relationship between three properties of test suites: size, structural coverage, and fault-finding effectiveness. In particular, we study the question of whether achieving high coverage leads directly to greater effectiveness, or only indirectly through forcing a test suite to be larger. Our experiments indicate that coverage is sometimes correlated with effectiveness when size is controlled for, and that using both size and coverage yields a more accurate prediction of effectiveness than size alone. This in turn suggests that both size and coverage are important to test suite effectiveness. Our experiments also indicate that no linear relationship exists among the three variables of size, coverage and effectiveness, but that a nonlinear relationship does exist.

The use of mutation in testing experiments and its sensitivity to external threats

Mutation analysts has emerged as a standard approach for empirical assessment of testing techniques. The test practitioners decide about cost-effectiveness of testing strategies based on the number of mutants the testing techniques detect. Though fundamental rigor to empirical software testing, the use of mutants in the absence of real-world faults has raised the concern of whether mutants and real faults exhibit similar properties. This paper revisits this important concern and disseminates interesting findings regarding mutants and whether these synthetic faults can predict fault detection ability of test suites. The results of controlled experiments conducted in this paper show that mutation when used in testing experiments is highly sensitive to external threats caused by some influential factors including mutation operators, test suite size, and programming languages. This paper raises the awareness message of the use of mutation in testing experiment and suggests that any interpretation or generalization of experimental findings based on mutation should be justified according to the influential factors involved.

Trimming Test Suites with Coincidentally Correct Test Cases for Enhancing Fault Localizations

Although empirical studies have demonstrated the usefulness of statistical fault localizations based on code coverage, the effectiveness of these techniques may be deteriorated due to the presence of some undesired circumstances such as the existence of coincidental correctness where one or more passing test cases exercise a faulty statement and thus causing some confusion to decide whether the underlying exercised statement is faulty or not. Fault localizations based on coverage can be improved if all possible instances of coincidental correctness are identified and proper strategies are employed to deal with these troublesome test cases. We introduce a technique to effectively identify coincidentally correct test cases. The proposed technique combines support vector machines and ensemble learning to detect mislabeled test cases, i.e. Coincidentally correct test cases. The ensemble-based support vector machine then can be used to trim a test suite or flip the test status of the coincidental correctness test cases and thus improving the effectiveness of fault localizations.

Prioritizing Mutation Operators Based on Importance Sampling

Mutation testing is a fault-based testing technique for measuring the adequacy of a test suite. Test suites are assigned scores based on their ability to expose synthetic faults (i.e., mutants) generated by a range of well-defined mathematical operators. The test suites can then be augmented to expose the mutants that remain undetected and are not semantically equivalent to the original code. However, the mutation score can be increased superfluously by mutants that are easy to expose. In addition, it is infeasible to examine all the mutants generated by a large set of mutation operators. Existing approaches have therefore focused on determining the sufficient set of mutation operators and the set of equivalent mutants. Instead, this paper proposes a novel Bayesian approach that prioritizes operators whose mutants are likely to remain unexposed by the existing test suites. Probabilistic sampling methods are adapted to iteratively examine a subset of the available mutants and direct focus towards the more informative operators. Experimental results show that the proposed approach identifies more than 90% of the important operators by examining ? 20% of the available mutants, and causes a 6% increase in the importance measure of the selected mutants.

Identifying Effective Test Cases through K-Means Clustering for Enhancing Regression Testing

Testing is the most time consuming and expensive process in the software development life cycle. In order to reduce the cost of regression testing, we propose a test case classification methodology based on k-means clustering with the purpose of classifying test cases into two groups of effective and non-effective test cases. The clustering strategy is based on Hamming distances measured over the differences between coverage information obtained for current and the previous releases of the program under test. Our empirical study shows that the clustering-based test case classification can identify effective test cases with high recall ratio and considerable accuracy percentage. The paper also investigates and compares the performance of the proposed clustering-based approach with various factors including coverage criteria and the weights factor used in measuring distances.

How Significant is the Effect of Fault Interactions on Coverage-Based Fault Localizations?

The effectiveness of coverage-based fault localizations in the presence of multiple faults has been a major concern for the software testing research community. A commonly held belief is that the fault localization techniques based on coverage statistics are less effective in the presence of multiple faults and their performance deteriorates. The fault interference phenomenon refers to cases where the software under test contains multiple faults whose interactions hinder effective debugging. The immediate research question that arises is to what extent fault interactions are influential. This paper focuses on verifying the existence of fault interference phenomenon in programs developed in programming languages with object-oriented features. The paper then statistically measures the influence and significance of fault interactions on the performance of debugging based on coverage-based fault localizations. The result verifies that the fault interleaving phenomenon occurs. However, its impact on the performance of fault localizations is negligible.

Bayesian reasoning for software testing

Despite significant advances in software testing research, the ability to produce reliable software products for a variety of critical applications remains an open problem. The key challenge has been the fact that each program or software product is unique, and existing methods are predominantly not capable of adapting to the observations made during program analysis. This paper makes the following claim: Bayesian reasoning methods provide an ideal research paradigm for achieving reliable and efficient software testing and program analysis. A brief overview of some popular Bayesian reasoning methods is provided, along with a justification of why they are applicable to software testing. Furthermore, some practical challenges to the widespread use of Bayesian methods are discussed, along with possible solutions to these challenges.

Predicting Vulnerable Software Components through N-Gram Analysis and Statistical Feature Selection

Vulnerabilities need to be detected and removed from software. Although previous studies demonstrated the usefulness of employing prediction techniques in deciding about vulnerabilities of software components, the accuracy and improvement of effectiveness of these prediction techniques is still a grand challenging research question. This paper proposes a hybrid technique based on combining N-gram analysis and feature selection algorithms for predicting vulnerable software components where features are defined as continuous sequences of token in source code files, i.e., Java class file. Machine learning-based feature selection algorithms are then employed to reduce the feature and search space. We evaluated the proposed technique based on some Java Android applications, and the results demonstrated that the proposed technique could predict vulnerable classes, i.e., software components, with high precision, accuracy and recall.

Feature Selections for Effectively Localizing Faulty Events in GUI Applications

Due to the complex causality of failure and the special characteristics of test cases, the faults in GUI (Graphic User Interface) applications are difficult to localize. This paper adapts feature selection algorithms to localize GUI-related faults in a given program. Features are defined as the subsequences of events executed. By employing statistical feature ranking techniques, the events can be ranked by the suspiciousness of events being responsible to exhibit faulty behavior. The features defined in a given source code implementing (event handle) the underlying event are then ranked in suspiciousness order. The evaluation of the proposed technique based on some open source Java projects verified the effectiveness of this feature selection based fault localization technique for GUI applications.