Dimitris E. Simos

On the applicability of combinatorial testing to web application security testing: a case study

Case studies for evaluating tools in security testing are powerful. Although they cannot achieve the scientific rigor of formal experiments, the results can provide sufficient information to help professionals judge if a specific technology being evaluated will benefit their organization. This paper reports on a case study done for evaluating and revisiting a recently introduced combinatorial testing methodology used for web application security purposes. It further reports on undertaken practical experiments thus strengthening the applicability of combinatorial testing to web application security testing.

Attack pattern-based combinatorial testing

The number of potential security threats rises with the increasing number of web applications, which cause tremendous financial and existential implications for developers and users as well. The biggest challenge for security testing is to specify and implement ways in order to detect potential vulnerabilities of the developed system in a never ending quest against new security threats but also to cover already known ones so that a program is suited against typical attack vectors. For these purposes many approaches have been developed in the area of model-based security testing in order to come up with solutions for real-world application problems. These approaches provide theoretical background as well as practical solutions for certain security issues. In this paper, we partially rely on previous work but focus on the representation of attack patterns using UML state diagrams. We extend previous work in combining the attack pattern models with combinatorial testing in order to provide concrete test input, which is submitted to the system under test. With combinatorial testing we capture different combinations of inputs and thus increasing the likelihood to find weaknesses in the implementation under test that can be exploited. Besides the foundations of our approach we further report on first experiments that indicate its practical use.

Attack Pattern-Based Combinatorial Testing with Constraints for Web Security Testing

Security testing of web applications remains a major problem of software engineering. In order to reveal vulnerabilities, manual and automatic testing approaches use different strategies for detection of certain kinds of inputs that might lead to a security breach. In this paper we compared a state-of-the-art manual testing tool with an automated one that is based on model-based testing. The first tool requires user input from the tester whereas the second one reduces the necessary amount of manual manipulation. Both approaches depend on the corresponding test case generation technique and its produced inputs are executed against the system under test (SUT). For this case we enhance a novel technique, which combines a combinatorial testing technique for input generation and a model-based technique for test execution. In this work the input parameter modelling is improved by adding constraints to generate more comprehensive and sophisticated testing inputs. The evaluated results indicate that both techniques succeed in detecting security leaks in web applications with different results, depending on the background logic of the testing approach. Last but not least, we claim that attack pattern-based combinatorial testing with constraints can be an alternative method for web application security testing, especially when we compare our method to other test generation techniques like fuzz testing.

Eris: A Tool for Combinatorial Testing of the Linux System Call Interface

In this paper, we show the applicability of combinatorial testing to the system call interface of the Linux kernel. Our approach is two-fold: first we analyze the Trinity fuzz tester and in the aftermath we adapt the input parameter modeling of Trinity to the field of combinatorial testing. Furthermore, apart from the modeling itself, we target to provide a configurable testing framework for executing tests obtained by the ACTS combinatorial test generation tool, called Eris.

Evaluation of the IPO-Family algorithms for test case generation in web security testing

Security testing of web applications remains a major problem of software engineering. In order to reveal vulnerabilities, testing approaches use different strategies for detection of certain kinds of inputs that might lead to a security breach. Such approaches depend on the corresponding test case generation technique that are executed against the system under test. In this work we examine how two of the most popular algorithms for combinatorial test case generation, namely the IPOG and IPOG-F algorithms, perform in web security testing. For generating comprehensive and sophisticated testing inputs we have used input parameter modelling which includes also constraints between the different parameter values. To handle the test execution, we make use of a recently introduced methodology which is based on model-based testing. Our evaluation indicates that both algorithms generate test inputs that succeed in revealing security leaks in web applications with IPOG-F giving overall slightly better results w.r.t. the test quality of the generated inputs. In addition, using constraints during the modelling of the attack grammars results in an increase on the number of test inputs that cause security breaches. Last but not least, a detailed analysis of our evaluation results confirms that combinatorial testing is an efficient test case generation method for web security testing as the security leaks are mainly due to the interaction of a few parameters. This statement is further supported by some combinatorial coverage measurement experiments on the successful test inputs.

Testing TLS Using Combinatorial Methods and Execution Framework

The TLS protocol is the standard for secure Internet communication between two parties. Unfortunately, there have been recently successful attacks like DROWN or BREACH that indicate the necessity for thoroughly testing TLS implementations. In our research work, we focus on automated test case generation and execution for the TLS security protocol, where the aim is to make use of combinatorial methods for providing test cases that ideally also reveal previously unknown attacks. This is made feasible by creating appropriate input parameter models for different messages that can appear in a TLS message sequence. In this paper, we present the resulting test case generation and execution framework together with the corresponding testing oracle. Furthermore, we discuss first empirical results obtained using different TLS implementations and their releases.

Coveringcerts: Combinatorial Methods for X.509 Certificate Testing

Correct behaviour of X.509 certificate validation code in SSL/TLS implementations is crucial to ensure secure communication channels. Recently, there have been major efforts in testing these implementations, namely frankencerts and mucerts, which provide new ways to generate test certificates which are likely to reveal errors in the implementations of X.509 validation logic. However, it remains a significant challenge to generate effective test certificates. In this paper, we explore the applicability of a prominent combinatorial method, namely combinatorial testing, for testing of X.509 certificates. We demonstrate that combinatorial testing provides the theoretical guarantees for revealing errors in the certificate validation logic of SSL/TLS implementations. Our findings indicate that the introduced combinatorial testing constructs, coveringcerts, compare favorably to existing testing methods by encapsulating the semantics of the validation logic in the input model and employing combinatorial strategies that significantly reduce the number of tests needed. Besides the foundations of our approach, we also report on experiments that indicate its practical use.

Planning-Based Security Testing of the SSL/TLS Protocol

With a growing amount of transferred data in an interconnected world, the insurance of a secure communication between two peers becomes a critical task in the software industry. A leak of critical data can cause tremendous costs in a financial, social but also political manner. For this sake, cryptographic protocols are implemented and regulate the data transfer, thus ensuring the safety of transferred data between two peers. The widespread security protocol SSL/TLS provides the mechanisms for this request, however, not without drawbacks since several security leaks have been identified up to now. Since vulnerabilities act as a starting point for a potential malicious action, the identification of such leaks is of highest priority. In this paper a novel testing approach is presented, which adapts planning for security testing of cryptographic protocols. The whole approach is implemented in one testing framework. Its purpose is to automatically test for known vulnerabilities in protocol implementations but to trigger other unintended behavior as well so eventually new security flaws can be identified. Additionally, the planning specification can be extended further so new testing possibilities can be generated. New test cases can be generated dynamically according to changing conditions.

A Combinatorial Approach to Analyzing Cross-Site Scripting (XSS) Vulnerabilities in Web Application Security Testing

Web applications typically employ sanitization functions to sanitize user inputs, independently whether this input is assumed to be legitimate, invalid or malicious. When such functions do not work correctly, a web application immediately becomes vulnerable to security attacks such as XSS. In this paper, we report a combinatorial approach to analyze XSS vulnerabilities in web applications. Our approach first performs combinatorial testing where a set of test vectors is executed against a subject application. If one or more XSS vulnerabilities are triggered during testing, we analyze the structure of each test vector to identify XSS-inducing combinations of its parameter model. If an attack vector contains an XSS-inducing combination, then the execution of this vector will successfully exploit an XSS vulnerability. Identification of XSS-inducing combinations provides insights about which kinds of user input might still be leverageable for XSS attacks and how to correct the function to provide better security guarantees. We conducted an experiment in which our approach was applied to four sanitization functions from the Web Application Vulnerability Scanner Evaluation Project (WAVSEP). The experimental results show that our approach can effectively identify XSS-inducing combinations for these sanitization functions.

Set-Based Algorithms for Combinatorial Test Set Generation

Testing is an important and expensive part of software and hardware development. Over the recent years, the construction of combinatorial interaction tests rose to play an important role towards making the cost of testing more efficient. Covering arrays are the key element of combinatorial interaction testing and a means to provide abstract test sets. In this paper, we present a family of set-based algorithms for generating covering arrays and thus combinatorial test sets. Our algorithms build upon an existing mathematical method for constructing independent families of sets, which we extend sufficiently in terms of algorithmic design in this paper. We compare our algorithms against commonly used greedy methods for producing 3-way combinatorial test sets, and these initial evaluation results favor our approach in terms of generating smaller test sets.