Rick Kuhn

Practical Combinatorial Testing: Beyond Pairwise

With new algorithms and tools, developers can apply high-strength combinatorial testing to detect elusive failures that occur only when multiple components interact. In pairwise testing, all possible pairs of parameter values are covered by at least one test, and good tools are available to generate arrays with the value pairs. In the past few years, advances in covering-array algorithms, integrated with model checking or other testing approaches, have made it practical to extend combinatorial testing beyond pairwise tests. The US National Institute of Standards and Technology (NIST) and the University of Texas, Arlington, are now distributing freely available methods and tools for constructing large t-way combination test sets (known as covering arrays), converting covering arrays into executable tests, and automatically generating test oracles using model checking (http://csrc.nist.gov/acts). In this review, we focus on real-world problems and empirical results from applying these methods and tools.

Combinatorial Software Testing

Combinatorial testing can detect hard-to-find software faults more efficiently than manual test case selection methods. While the most basic form of combinatorial testing-pairwise-is well established, and adoption by software testing practitioners continues to increase, industry usage of these methods remains patchy at best. However, the additional training required is well worth the effort.

Data Loss Prevention

In todays digital economy, data enters and leaves cyberspace at record rates. A typical enterprise sends and receives millions of email messages and downloads, saves, and transfers thousands of files via various channels on a daily basis. Enterprises also hold sensitive data that customers, business partners, regulators, and shareholders expect them to protect. Unfortunately, companies constantly fall victim to massive data loss, and high-profile data leakages involving sensitive personal and corporate data continue to appear (http://opensecurityfoundation. org). Data loss could substantially harm a companys competitiveness and reputation and could also invite lawsuits or regulatory consequences for lax security. Therefore, organizations should take measures to understand the sensitive data they hold, how its controlled, and how to prevent it from being leaked or compromised.

A combinatorial approach to building navigation graphs for dynamic web applications

Modeling the navigation structure of a dynamic web application is a challenging task because of the presence of dynamic pages. In particular, there are two problems to be dealt with: (1) the page explosion problem, i.e., the number of dynamic pages may be huge or even infinite; and (2) the request generation problem, i.e., many dynamic pages may not be reached unless appropriate user requests are supplied. As a user request typically consists of multiple parameter values, the request generation problem can be further divided into two problems: (1) How to select appropriate values for individual parameters? (2) How to effectively combine individual parameter values to generate requests? This paper presents a combinatorial approach to building a navigation graph. The novelty of our approach is two-fold. First, we use an abstraction scheme to control the page explosion problem. In this scheme, pages that are likely to have the same navigation behavior are grouped together, and are represented as a single node in a navigation graph. Grouping pages reduces and bounds the size of a navigation graph for practical applications. Second, assuming that values of individual parameters are supplied by using other techniques or generated manually by the user, we combine parameter values in a way that achieves a well-defined combinatorial coverage called pairwise coverage. Using pairwise coverage can significantly reduce the number of requests that have to be submitted while still achieving effective coverage of the navigation structure. We report a prototype tool called Tansuo, and apply the tool to five open source web applications. Our empirical results indicate that Tansuo can efficiently generate web navigation graphs for these applications.

Combinatorial Testing of ACTS: A Case Study

In this paper we present a case study of applying combinatorial testing to test a combinatorial test generation tool called ACTS. The purpose of this study is two-fold. First, we want to gain experience and insights about how to apply combinatorial testing in practice. Second, we want to evaluate the effectiveness of combinatorial testing applied to a real-life system. ACTS has 24637 lines of uncommented code, and provides a command line interface and a fairly sophisticated graphic user interface. The main challenge of this study was to model the input space in terms of a set of parameters and values. Once the model was designed, we generated test cases using ACTS, which were then later used to test ACTS. The results of this study show that input space modeling can be a significant undertaking, and needs to be carefully managed. The results also show that combinatorial testing is effective in terms of achieving high code coverage and fault detection.

A combinatorial approach to detecting buffer overflow vulnerabilities

Buffer overflow vulnerabilities are program defects that can cause a buffer to overflow at runtime. Many security attacks exploit buffer overflow vulnerabilities to compromise critical data structures. In this paper, we present a black-box testing approach to detecting buffer overflow vulnerabilities. Our approach is motivated by a reflection on how buffer overflow vulnerabilities are exploited in practice. In most cases the attacker can influence the behavior of a target system only by controlling its external parameters. Therefore, launching a successful attack often amounts to a clever way of tweaking the values of external parameters. We simulate the process performed by the attacker, but in a more systematic manner. A novel aspect of our approach is that it adapts a general software testing technique called combinatorial testing to the domain of security testing. In particular, our approach exploits the fact that combinatorial testing often achieves a high level of code coverage. We have implemented our approach in a prototype tool called Tance. The results of applying Tance to five open-source programs show that our approach can be very effective in detecting buffer overflow vulnerabilities.

Vetting Mobile Apps

Billions of copies of apps for mobile devices have been purchased in recent years. With this growth, however, comes an increase in the spread of potentially dangerous security vulnerabilities. Because of an apps low cost and high proliferation, the threat of these vulnerabilities could be far greater than that of traditional computers. Thus, purchasing organizations or third-party labs should vet the apps before selling them, and consumers need to understand the risks of apps and the prospects for ensuring their security.

Extensible Access Control Markup Language (XACML) and Next Generation Access Control (NGAC)

Extensible Access Control Markup Language (XACML) and Next Generation Access Control (NGAC) are very different attribute based access control standards with similar goals and objectives. An objective of both is to provide a standardized way for expressing and enforcing vastly diverse access control policies in support of various types of data services. The two standards differ with respect to the manner in which access control policies and attributes are specified and managed, and decisions are computed and enforced. This paper is presented as a consolidation and refinement of public draft NIST SP 800-178 [21], describing, and comparing these two standards.

Surviving Insecure IT: Effective Patch Management

The amount of time to protect enterprise systems against potential vulnerability continues to shrink. Enterprises need an effective patch management mechanism to survive the insecure IT environment. Effective patch management is a systematic and repeatable patch distribution process which includes establishing timely and practical alerts, receiving notification of patches or discovering them, downloading patches and documentation, assessing and prioritizing vulnerabilities, performing testing, deploying patches, and auditing.

Introducing Combinatorial Testing in a Large Organization: Pilot Project Experience Report

This poster gives an overview of the experience of eight pilot projects, over two years, applying combinatorial testing in Lockheed Martin (LM), one of the worlds largest aerospace firms. Lockheed Martin and NIST developed a Co-operative Research and Development Agreement (CRADA) to evaluate effectiveness and areas of suitable application for combinatorial testing in a real-world industrial setting with complex software requirements. (One of the ways in which NIST conducts joint research with US industry is through CRADAs, which allow federal laboratories to work with US industry and provide flexibility in structuring projects, intellectual property rights, and in protecting industry proprietary information and research results).