Vadim Okun

IPOG: A General Strategy for T-Way Software Testing

Most existing work on t-way testing has focused on 2-way (or pairwise) testing, which aims to detect faults caused by interactions between any two parameters. However, faults can also be caused by interactions involving more than two parameters. In this paper, we generalize an existing strategy, called in-parameter-order (IPO), from pairwise testing to t-way testing. A major challenge of our generalization effort is dealing with the combinatorial growth in the number of combinations of parameter values. We describe a t-way testing tool, called FireEye, and discuss design decisions that are made to enable an efficient implementation of the generalized IPO strategy. We also report several experiments that are designed to evaluate the effectiveness of FireEye

IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing

Electrical harness manufacturing apparatus comprises wire feeding means for feeding wires along a wire feed path through upstream and downstream (relative to the direction of wire feed) wire guides. The guides have opposed ends which are adjacent to each other during feeding. The guides thereafter move apart so that fed wires are exposed in a gap between the opposed ends. A transferring device clamps the wires in the gap and wire cutting means are provided to cut the wires adjacent to the transferring means, thereby producing leads having their trailing ends gripped in the transferring means. The transferring means transfers the trailing ends laterally of the feed path to a wire connecting station at which the trailing ends are connected to terminals in a connector. Insulation can be stripped, if desired, from the trailing ends of the cut leads and from the leading ends of the wires extending from the feed means.

Mutation operators for specifications

Testing has a vital support role in the software engineering process, but developing tests often takes significant resources. A formal specification is a repository of knowledge about a system, and a recent method uses such specifications to automatically generate complete test suites via mutation analysis. We define an extensive set of mutation operators for use with this method. We report the results of our theoretical and experimental investigation of the relationships between the classes of faults detected by the various operators. Finally, we recommend sets of mutation operators which yield good test coverage at a reduced cost compared to using all proposed operators.

Web Application Scanners: Definitions and Functions

There are many commercial software security assurance tools that claim to detect and prevent vulnerabilities in application software. However, a closer look at the tools often leaves one wondering which tools find what vulnerabilities. This paper identifies a taxonomy of software security assurance tools and defines one type of tool: Web application scanner, i.e., an automated program that examines Web applications for security vulnerabilities. We describe the types of functions that are generally found in a Web application scanner and how to test it

Comparison of fault classes in specification-based testing

Abstract Our results extending Kuhns fault class hierarchy provide a justification for the focus of fault-based testing strategies on detecting particular faults and ignoring others. We develop a novel analytical technique which allows us to elegantly prove that the hierarchy applies to arbitrary expressions, not just those in disjunctive normal form. We also use the technique to extend the hierarchy to a wider range of fault classes. To demonstrate broad applicability, we compare faults in practical situations and analyze previous results. In particular, using our technique, we show that the basic meaningful impact strategy of Weyuker et al. tests for stuck-at faults, not just variable negation faults.

Mutation of model checker specifications for test generation and evaluation

Mutation analysis on model checking specifications is a recent development. This approach mutates a specification, then applies a model checker to compare the mutants with the original specification to automatically generate tests or evaluate coverage. The properties of specification mutation operators have not been explored in depth. We report our work on theoretical and empirical comparison of these operators. Our future plans include studying how the form of a specification influences the results, finding relations between different operators, and validating the method against independent metrics.

Effect of static analysis tools on software security: preliminary investigation

Static analysis tools can handle large-scale software and find thousands of defects. But do they improve software security? We evaluate the effect of static analysis tool use on software security in open source projects. We measure security by vulnerability reports in the National Vulnerability Database.

Evaluating bug finders: test and measurement of static code analyzers

Software static analysis is one of many options for finding bugs in software. Like compilers, static analyzers take a program as input. This paper covers tools that examine source code - without executing it - and output bug reports. Static analysis is a complex and generally undecidable problem. Most tools resort to approximation to overcome these obstacles and it sometimes leads to incorrect results. Therefore, tool effectiveness needs to be evaluated. Several characteristics of the tools should be examined. First, what types of bugs can they find? Second, what proportion of bugs do they report? Third, what percentage of findings is correct? These questions can be answered by one or more metrics. But to calculate these, we need test cases having certain characteristics: statistical significance, ground truth, and relevance. Test cases with all three attributes are out of reach, but we can use combinations of only two to calculate the metrics. The results in this paper were collected during Static Analysis Tool Exposition (SATE) V, where participants ran 14 static analyzers on the test sets we provided and submitted their reports to us for analysis. Tools had considerably different support for most bug classes. Some tools discovered significantly more bugs than others or generated mostly accurate warnings, while others reported wrong findings more frequently. Using the metrics, an evaluator can compare candidates and select the tool that aligns best with his or her objectives. In addition, our results confirm that the bugs most commonly found by tools are among the most common and important bugs in software. We also observed that code complexity is a major hindrance for static analyzers and detailed which code constructs tools handle well and which impede their analysis.

Of Massive Static Analysis Data

The Software Assurance Metrics and Tool Evaluation (SAMATE) project at the National Institute of Standards and Technology (NIST) has organized four Static Analysis Tool Expositions (SATE). SATE is designed to advance research in static analysis tools that find security-relevant defects in source code. Briefly, participating tool makers run their tools on a set of programs. Researchers led by NIST analyze the tool outputs. The results and experiences are reported at a workshop. These expositions have accumulated large amounts of data. This collection allowed for the development and validation of practical metrics in regard to static analysis tool effectiveness and independence. In this paper, we discuss the role of the data in determining which metrics can be derived. Specifically, we detail the three characteristics test data should exhibit and explain why the data we use express each combination of two out of these three properties.