Andrea Calvagna

A logic-based approach to combinatorial testing with constraints

Usage of combinatorial testing is wide spreading as an effective technique to reveal unintended feature interaction inside a given system. To this aim, test cases are constructed by combining tuples of assignments of the different input parameters, based on some effective combinatorial strategy. The most commonly used strategy is two-way (pairwise) coverage, requiring all combinations of valid assignments for all possible pairs of input parameters to be covered by at least one test case. In this paper a new heuristic strategy developed for the construction of pairwise covering test suites is presented, featuring a new approach to support expressive constraining over the input domain. Moreover, it allows the inclusion or exclusion of ad-hoc combinations of parameter bindings to let the user customize the test suite outcome. Our approach is tightly integrated with formal logic, since it uses test predicates to formalize combinatorial testing as a logic problem, and applies an external model checker tool to solve it. The proposed approach is supported by a prototype tool implementation, and early results of experimental assessment are also presented.

Expeerience: a JXTA middleware for mobile ad-hoc networks

The combination of personal computing devices and wireless ad-hoc networks allows the concept of mobile ad-hoc information system, consisting of a highly dynamic, decentralized and self-organizing network of autonomous and mobile devices that interact as peers [M. Hannicainen et al. (2002), L. Kleinrock, (2000)]. Here we define a middleware layer, named Expeerience. We have decided to adopt as a basic software framework, an emerging P2P open technology such as JXTA. The second issue we address in this work is the integration of some code mobility support in the developed middleware to allow the distribution and execution of services on peers that originally do not own the service code. This has been introduced in our platform by adding a mobile code service, which enables to download and install new services dynamically only if necessary.

IPO-s: Incremental Generation of Combinatorial Interaction Test Data Based on Symmetries of Covering Arrays

Veri¿cation and validation of highly-con¿gurable software systems, such as those supporting many optional or customizable features, is a challenging activity. In fact, due to its intrinsic complexity, formal modeling of the whole system may require a great effort. Modeling activities may become extremely expensive and time consuming, and the tester may decide to model (at least initially) only the inputs and require they are suf¿ciently covered by tests. The recent wide spreading usage of combinatorial interaction testing (CIT) is dramatically improving the effectiveness of this activity. Although there exist analytical ways to derive minimal sized CIT test suites, they are not applicable to all task sizes. Therefore, researchers have explored many techniques based on greedy or heuristic algorithms that may lead to sub-optimal result in the size of the built test suite but that are applicable to problems of real size. In this paper, a new parameter-based heuristic algorithm for the construction of pairwise covering test suites is presented; it is based on a symmetry property of covering arrays and it is called IPOS . Time and space complexity of IPOS is discussed in comparison especially with the only other parameter-based approach existing in literature. The proposed approach is supported by a prototype implementation, and experimental assessment is also presented.

A Formal Logic Approach to Constrained Combinatorial Testing

Combinatorial testing is as an effective testing technique to reveal failures in a given system, based on input combinations coverage and combinatorial optimization. Combinatorial testing of strengtht (t ≥ 2) requires that each t-wise tuple of values of the different system input parameters is covered by at least one test case. Combinatorial test suite generation algorithms aim at producing a test suite covering all the required tuples in a small (possibly minimal) number of test cases, in order to reduce the cost of testing. The most used combinatorial technique is the pairwise testing (t = 2) which requires coverage of all pairs of input values. Constrained combinatorial testing takes also into account constraints over the system parameters, for instance forbidden tuples of inputs, modeling invalid or not realizable input values combinations. In this paper a new approach to combinatorial testing, tightly integrated with formal logic, is presented. In this approach, test predicates are used to formalize combinatorial testing as a logical problem, and an external formal logic tool is applied to solve it. Constraints over the input domain are expressed as logical predicates too, and effectively handled by the same tool. Moreover, inclusion or exclusion of select tuples is supported, allowing the user to customize the test suite layout. The proposed approach is supported by a prototype tool implementation and results of experimental assessment are also presented.

T-wise combinatorial interaction test suites construction based on coverage inheritance

Combinatorial interaction testing (CIT) is a testing technique that requires covering all t‐sized tuples of values out of n parameter attributes or properties modelled after the input parameters or the configuration domain of a system under test. CIT test suites have shown to be very effective in software testing already at pairwise (t  =  2) level, and the effectiveness of CIT grows with the tuple width t. Unfortunately, the number of tuples to be tested also does grow. In order to reduce the testing effort, researchers addressed the issue of computing minimal‐sized CIT test suites with effective and scalable algorithms. However, still very few generally applicable t‐wise covering construction algorithms (and tools) do exist in literature. This paper presents an original greedy algorithm to compute t‐wise covering mixed covering arrays with constant space complexity, irrespective of the number of involved parameters and strength of interaction. The proposed algorithm has been implemented in a prototype tool, featuring also support for user constraints over the inputs. Assessment of the tool performance on a set of large, real‐world test systems is reported, with results encouraging its adoption in industrial production environments. Copyright

Combining Satisfiability Solving and Heuristics to Constrained Combinatorial Interaction Testing

Combinatorial interaction testing aims at revealing errors inside a system under test triggered by unintended interaction between values of its input parameters. In this context we defined a new greedy approach to generate a combinatorial interaction test suites in the presence of constraints, based on integration of an SMT solver, and ordered processing of test goals. Based on the observation that the processing order of required combinations determines the size of the final test suite, this approach has been then used as a framework to evaluate a set of deterministic ordering strategies, each based on a different heuristic optimization criteria. Their performance has been assessed and contrasted also with those of random and dummy ordering strategies. Results of experimental assessment are presented and compared with well-known combinatorial tools.

Combinatorial Interaction Testing with CITLAB

In this paper the CITLAB tool for Combinatorial Interaction Testing is presented. The tool allows importing/exporting models of combinatorial problems from/to different application domains, by means of a common interchange syntax notation and a corresponding interoperable semantic metamodel. Moreover, the tool is a framework allowing embedding and transparent invocation of multiple, different implementations of combinatorial algorithms. CITLAB has been designed tightly integrated with the Eclipse IDE framework, by means of its plug-in extension mechanism. It is intended to easy the spread of CIT testing both in industrial practice and in academic research, by allowing users and researchers to apply multiple test suite generation algorithms, each with its peculiarities, on the same problem models, and let them compare the results in order to select the one that best fits their needs, while alleviating from the pain of knowing all the different details and notations of the underlying CIT tools.

Delivering Dependable Reusable Components by Expressing and Enforcing Design Decisions

A component is usually complemented with guidelines expressing its proper use, e.g. the appropriate order of calls that clients should conform to. During reuse, clients could easily alter such an order, and this could result in reused components that become unreliable, since architectural guidelines have not been honoured. Sometimes architectural guidelines are simply unknown, hence whether components are misused by clients is uncertain. This paper proposes an approach to document the architectural guidelines that client classes should comply with when reusing a component. We empower component developers to provide such guidelines, conveying design decisions, along with the code of components in such a way to be apt to automatic checks. Then, clients compliance with architectural guidelines of reused components can be automatically checked by an aspect-based tool. As a result, proper usage of reused components can be ensured, and in turn the behaviour of components should be correct. This strengthen the reliability of the resulting system. Especially for continuous evolution, having automatic conformance checks is paramount for obtaining the correct behaviour of reused components.

Combinatorial Testing for Feature Models Using CitLab

Feature models are commonly used to represent product lines and systems with a set of features interrelated each others. Test generation from feature models, i.e. generating a valid and representative subset of all the possible product configurations, is still an open challenge. A common approach is to build combinatorial interaction test suites, for instance achieving pairwise coverage among the features. In this paper we show how standard feature models can be translated to combinatorial interaction models in our framework CITLAB, with all the advantages of having a combinatorial testing environment (in terms of a clear semantics, editing facilities, language for seeds and test goals, and generation algorithms). We present our translation which gives a precise semantics to feature models and it tries to minimize the number of parameter and constraints while preserving the original semantics of the feature model. Experiments show the advantages of our approach.

Mobility and quality of service across heterogeneous wireless networks

The ability to guarantee Quality of Service (QoS) is one of the key issues in the creation of a telecommunication system. In this paper, we deal with some aspects of providing QoS in wired cum wireless communication environment, focusing on the specific feature of guaranteeing session continuity when the wireless networks, to which a mobile terminal connects to, are also heterogeneous. The terms that define what QoS is should, in this case, be reviewed in order to account also for all aspects related to user mobility inside such an environment. In this paper, such a new framework for the representation of QoS is proposed. Moreover, a wireless mobility test case is also presented. This is based on appropriate middleware we implemented, which allows a mobile terminal to experience real wireless IP mobility while moving on a large spatial scale between different sites, scattered over a metropolitan area.