Rubing Huang

Adaptive Random Test Case Generation for Combinatorial Testing

Random testing (RT), a fundamental software testing technique, has been widely used in practice. Adaptive random testing (ART), an enhancement of RT, performs better than original RT in terms of fault detection capability. However, not much work has been done on effectiveness analysis of ART in the combinatorial test spaces. In this paper, we propose a novel family of ART-based algorithms for generating combinatorial test suites, mainly based on fixed-size-candidate-set ART and restricted random testing (that is, ART by exclusion). We use an empirical approach to compare the effectiveness of test sets obtained by our proposed methods and random selection strategy. Experimental data demonstrate that the ART-based tests cover all possible combinations at a given strength more quickly than randomly chosen tests, and often detect more failures earlier and with fewer test cases in simulations.

Prioritizing Variable-Strength Covering Array

Combinatorial interaction testing is a well-studied testing strategy, and has been widely applied in practice. Combinatorial interaction test suite, such as fixed-strength and variable-strength interaction test suite, is widely used for combinatorial interaction testing. Due to constrained testing resources in some applications, for example in combinatorial interaction regression testing, prioritization of combinatorial interaction test suite has been proposed to improve the efficiency of testing. However, nearly all prioritization techniques may only support fixed-strength interaction test suite rather than variable-strength interaction test suite. In this paper, we propose two heuristic methods in order to prioritize variable-strength interaction test suite by taking advantage of its special characteristics. The experimental results show that our methods are more effective for variable-strength interaction test suite by comparing with the technique of prioritizing combinatorial interaction test suites according to test case generation order, the random test prioritization technique, and the fixed-strength interaction test suite prioritization technique. Besides, our methods have additional advantages compared with the prioritization techniques for fixed-strength interaction test suite.

Prioritization of combinatorial test cases by incremental interaction coverage

Combinatorial testing is a well-recognized testing method, and has been widely applied in practice. To facilitate analysis, a common approach is to assume that all test cases in a combinatorial test suite have the same fault detection capability. However, when testing resources are limited, the order of executing the test cases is critical. To improve testing cost-effectiveness, prioritization of combinatorial test cases is employed. The most popular approach is based on interaction coverage, which prioritizes combinatorial test cases by repeatedly choosing an unexecuted test case that covers the largest number on uncovered parameter value combinations of a given strength (level of interaction among parameters). However, this approach suffers from some drawbacks. Based on previous observations that the majority of faults in practical systems can usually be triggered with parameter interactions of small strengths, we propose a new strategy of prioritizing combinatorial test cases by incrementally adjusting the strength values. Experimental results show that our method performs better than the random prioritization technique and the technique of prioritizing combinatorial test suites according to test case generation order, and has better performance than the interaction-coverage-based test prioritization technique in most cases.

Adaptive random prioritization for interaction test suites

Combinatorial interaction testing (CIT), a black-box testing method, has been well studied in recent years. It aims at constructing an effective interaction test suites, so as to identify the faults that are caused by interactions among parameters. After interaction test suites are generated by CIT, the execution order of test cases in the test suite becomes critical due to limited testing resources. To determine test case order, the prioritization of interaction test suites has been employed. As we know, random prioritization (RP) of test cases has been considered as simple but ineffective. Existing research unveils that adaptive random prioritization (ARP) of test cases is an alternative and promising candidate that may replace RP. However, previous ARP techniques may not be used to prioritize interaction test suites due to the lack of source-code-related information in interaction test suite, such as statement coverage, function coverage, or branch coverage. In this paper, we not only propose the ARP strategy in order to prioritize interaction test suites by using interaction coverage information, without the source-code-related information, but also unify the RP strategy and traditional interaction-coverage based prioritization strategy (ICBP). Additionally, simulation studies indicate that the ARP strategy performs better than the RP strategy, test-case-generation prioritization, and reverse test-case-generation prioritization, and can also be more time-saving than ICBP while greatly maintaining similar, or even better, effectiveness.

An approach of security testing for third-party component based on state mutation

It is essential to study an effective approach of security testing for third-party component. In this paper, to effectively trigger implicit vulnerabilities of third-party components, an approach of security testing for third-party component is proposed based on state mutation. To start with, executable method sequences of components are transformed into extended finite state machine. Then, according to characteristics of condition conflict and behavior conflict, two test case generation algorithms are addressed, that is, Operations Conflict Sequences Generation Algorithm and Conditions Conflict Sequences Generation Algorithm, which are designed to generate inaccessible sequences of behavior and condition conflicts. These conflict sequences are run. Furthermore, the security detecting algorithms are addressed to detect implicit vulnerabilities of third-party components, and then, testing report of component security is obtained. In the end, some experiments are conducted on the basis of the proposed approach, and the experimental results show that the proposed approach can effectively detect security exceptions of third-party components. Copyright

Worst-input mutation approach to web services vulnerability testing based on SOAP messages

The growing popularity and application of Web services have led to an increase in attention to the vulnerability of software based on these services. Vulnerability testing examines the trustworthiness, and reduces the security risks of software systems, however such testing of Web services has become increasing challenging due to the cross-platform and heterogeneous characteristics of their deployment. This paper proposes a worst-input mutation approach for testing Web service vulnerability based on SOAP (Simple Object Access Protocol) messages. Based on characteristics of the SOAP messages, the proposed approach uses the farthest neighbor concept to guide generation of the test suite. The test case generation algorithm is presented, and a prototype Web service vulnerability testing tool described. The tool was applied to the testing of Web services on the Internet, with experimental results indicating that the proposed approach, which found more vulnerability faults than other related approaches, is both practical and effective.

An effective long string searching algorithm towards component security testing

In the execution of method invocation sequences to test component security, abnormal or normal information is generated and recorded in a monitor log. By searching abnormal information from monitor log, the exceptions that the component has can be determined. To facilitate the searching process, string searching methods could be employed. However, current approaches are not effective enough to search long pattern string. In order to mine the specific information with less number of matches, we proposed an improved Sunday string searching algorithm in this paper. Unlike Sunday algorithm which does not make use of the already matched characters, the proposed approach presents two ideas - utilizing and recycling these characters. We take advantage of all matched characters in main string, if they are still in the matchable interval compared with pattern string, to increase the distance that pattern string moves backwards. Experimental analysis shows that, compared to Sunday algorithm, our method could greatly reduce the matching times, if the scale of character set constituting both main string and pattern string is small, or if the length of pattern string is long. Also, the proposed approach can improve the search effectiveness for abnormal information in component security testing.

A Similarity Metric for the Inputs of OO Programs and Its Application in Adaptive Random Testing

Random testing (RT) has been identified as one of the most popular testing techniques, due to its simplicity and ease of automation. Adaptive random testing (ART) has been proposed as an enhancement to RT, improving its fault-detection effectiveness by evenly spreading random test inputs across the input domain. To achieve the even spreading, ART makes use of distance measurements between consecutive inputs. However, due to the nature of object-oriented software (OOS), its distance measurement can be particularly challenging: Each input may involve multiple classes, and interaction of objects through method invocations. Two previous studies have reported on how to test OOS at a single-class level using ART. In this study, we propose a new similarity metric to enable multiclass level testing using ART. When generating test inputs (for multiple classes, a series of objects, and a sequence of method invocations), we use the similarity metric to calculate the distance between two series of objects, and between two sequences of method invocations. We integrate this metric with ART and apply it to a set of open-source OO programs, with the empirical results showing that our approach outperforms other RT and ART approaches in OOS testing.

Prioritizing Interaction Test Suites Using Repeated Base Choice Coverage

Combinatorial interaction testing is a well-studied testing strategy that aims at constructing an effective interaction test suite (ITS) of a specific generation strength to identify interaction faults caused by the interactions among factors. Due to limited testing resources in practice, for example in combinatorial interaction regression testing, interaction test suite prioritization (ITSP) has been proposed to improve the efficiency of testing. An intuitive ITSP strategy that has been widely used in practice is fixed-strength interaction coverage based prioritization (FICBP). FICBP makes use of a property of the ITS: interaction coverage at a fixed prioritization strength. However, a challenge facing FICBP is that, when the ITS is large, the prioritization cost can be very high. In this paper, we propose a new FICBP method that, by repeatedly using base choice coverage (i.e., one-wise coverage) during the prioritization process, improves testing efficiency while maintaining testing effectiveness. The empirical studies show that our method has fault detection capability comparable to current FICBP methods, but obtains more stable results in many cases. Additionally, our method requires considerably less prioritization time than other FICBP methods at different prioritization strengths.

An improved string-searching algorithm and its application in component security testing

Mass monitor logs are produced during the process of component security testing. In order to mine the explicit and implicit security exception information of the tested component, the log should be searched for keyword strings. However, existing string-searching algorithms are not very efficient or appropriate for the operation of searching monitor logs during component security testing. For mining abnormal information effectively in monitor logs, an improved string-searching algorithm is proposed. The main idea of this algorithm is to search for the first occurrence of a character in the main string. The character should be different and farther from the last character in the pattern string. With this algorithm, the backward moving distance of the pattern string will be increased and the matching time will be optimized. In the end, we conduct an experimental study based on our approach, the results of which show that the proposed algorithm finds strings in monitor logs 11.5% more efficiently than existing approaches.