Antonia Bertolino

Software Testing Research: Achievements, Challenges, Dreams

Software engineering comprehends several disciplines devoted to prevent and remedy malfunctions and to warrant adequate behaviour. Testing, the subject of this paper, is a widespread validation approach in industry, but it is still largely ad hoc, expensive, and unpredictably effective. Indeed, software testing is a broad term encompassing a variety of activities along the development cycle and beyond, aimed at different goals. Hence, software testing research faces a collection of challenges. A consistent roadmap of the most relevant challenges to be addressed is here proposed. In it, the starting point is constituted by some important past achievements, while the destination consists of four identified goals to which research ultimately tends, but which remain as unreachable as dreams. The routes from the achievements to the dreams are paved by the outstanding research challenges, which are discussed in the paper along with interesting ongoing work.

Automatic synthesis of behavior protocols for composable web-services

Web-services are broadly considered as an effective means to achieve interoperability between heterogeneous parties of a business process and offer an open platform for developing new composite web-services out of existing ones. In the literature many approaches have been proposed with the aim to automatically compose web-services. All of them assume that, along with the web-service signature, some information is provided about how clients interacting with the web-service should behave when invoking it. We call this piece of information the web-service behavior protocol. Unfortunately, in the practice this assumption turns out to be unfounded. To address this need, in this paper we propose a method to automatically derive from the web-service signature an automaton modeling its behavior protocol. The method, called StrawBerry, combines synthesis and testing techniques. In particular, synthesis is based on data type analysis. The conformance between the synthesized automaton and the implementation of the corresponding web-service is checked by means of testing. The application of StrawBerry to the Amazon E-Commerce Service shows that it is practical and realistic.

Using spanning sets for coverage testing

A test coverage criterion defines a set E/sub r/ of entities of the program flowgraph and requires that every entity in this set is covered under some test Case. Coverage criteria are also used to measure the adequacy of the executed test cases. In this paper, we introduce the notion of spanning sets of entities for coverage testing. A spanning set is a minimum subset of E/sub r/, such that a test suite covering the entities in this subset is guaranteed to cover every entity in E/sub r/. When the coverage of an entity always guarantees the coverage of another entity, the former is said to subsume the latter. Based on the subsumption relation between entities, we provide a generic algorithm to find spanning sets for control flow and data flow-based test coverage criteria. We suggest several useful applications of spanning sets: They help reduce and estimate the number of test cases needed to satisfy coverage criteria. We also empirically investigate how the use of spanning sets affects the fault detection effectiveness.

Automatic generation of path covers based on the control flow analysis of computer programs

Branch testing a program involves generating a set of paths that will cover every arc in the program flowgraph, called a path cover, and finding a set of program inputs that will execute every path in the path cover. This paper presents a generalized algorithm that finds a path cover for a given program flowgraph. The analysis is conducted on a reduced flowgraph, called a ddgraph, and uses graph theoretic principles differently than previous approaches. In particular, the relations of dominance and implication which form two trees of the arcs of the ddgraph are exploited. These relations make it possible to identify a subset of ddgraph arcs, called unconstrained arcs, having the property that a set of paths exercising all the unconstrained arcs also cover all the arcs in the ddgraph. In fact, the algorithm has been designed to cover all the unconstrained arcs of a given ddgraph: the paths are derived one at a time, each path covering at least one as yet uncovered unconstrained arc. The greatest merits of the algorithm are its simplicity and its flexibility. It consists in just visiting recursively in combination the dominator and the implied trees, and is flexible in the sense that it can derive a path cover to satisfy different requirements, according to the strategy adopted for the selection of the unconstrained arc to be covered at each recursive iteration. This feature of the algorithm can be employed to address the problem of infeasible paths, by adopting the most suitable selection strategy for the problem at hand. Embedding of the algorithm into a software analysis and testing tool is recommended. >

WS-TAXI: A WSDL-based Testing Tool for Web Services

Web Services (WSs) are the W3C-endorsed realization of the Service-Oriented Architecture (SOA). Since they are supposed to be implementation-neutral, WSs are typically tested black-box at their interface. Such an interface is generally specified in an XML-based notation called the WS Description Language (WSDL). Conceptually, these WSDL documents are eligible for fully automated WS test generation using syntax-based testing approaches. Towards such goal, we introduce the WS-TAXI framework, in which we combine the coverage of WS operations with data-driven test generation. In this paper we present an early-stage implementation of WS-TAXI, obtained by the integration of two existing softwares: soapUI, a popular tool for WS testing, and TAXI, an application we have previously developed for the automated derivation of XML instances from a XML schema. WS-TAXI delivers a complete suite of test messages ready for execution. Test generation is driven by basic coverage criteria and by the application of some heuristics. The application of WS-TAXI to a real case study gave encouraging results.

CB-SPE tool: putting component-based Performance Engineering into practice

A crucial issue in the design of Component-Based (CB) applications is the ability to early guarantee that the system under development will satisfy its Quality of Service requirements. In particular, we need rigorous and easy-to-use techniques for predicting and analyzing the performance of the assembly based on the properties of the constituent components. To this purpose, we propose the CB-SPE framework: a compositional methodology for CB Software Performance Engineering (SPE) and its supporting tool. CB-SPE is based on, and adapts to a CB paradigm, the concepts and steps of the well-known SPE technology, using for input modeling the standard RT-UML PA profile. The methodology is compositional: it is first applied by the component developer at the component layer, achieving a parametric performance evaluation of the components in isolation; then, at the application layer, the system assembler is provided with a step- wise procedure for predicting the performance of the assembled components on the actual platform. We have developed the CB-SPE tool reusing as much as possible existing free tools. In this paper we present the realized framework, together with a simple application example.

The Cow_Suite Approach to Planning and Deriving Test Suites in UML Projects

Cow_Suite provides an integrated and practical approach to the strategic generation and planning of UML-based test suites, since the early stages of system analysis and modeling. It consists of two original components working in combination: the Cowtest strategy, which organizes the testing process and helps the manager to select among the many potential test cases, and the UIT method, which performs the automated generation of test cases from the UML diagrams. The approach can be used in incremental way, starting from the preliminary (even incomplete) UML diagrams, and is applied to integration subsystems, as interactively selected by the tester. The emphasis is on usability, in that we use exactly the same UML diagrams developed for analysis and design, without requiring any additional formalism or ad-hoc effort specifically for testing purposes. Cow_Suite has been implemented in a prototype tool, and is currently being validated in an industrial development environment.

Use case-based testing of product lines

This paper presents PLUTO, a simple and intuitive methodology to manage the testing process of product lines, described as Product Lines Use Cases (PLUCs). PLUCs are an extension of the well-known Cockburns Use Cases, a notation based on natural language descriptions of requirements. The proposed test methodology is based on the Category Partition method, and can be used to derive a generic Test Specification for the product line, and a set of relevant test scenarios for a customer specific application.

Model-Based Generation of Testbeds for Web Services

A Web Service is commonly not an independent software entity, but plays a role in some business process. Hence, it depends on the services provided by external Web Services, to provide its own service. While developing and testing a Web Service, such external services are not always available, or their usage comes along with unwanted side effects like, e.g., utilization fees or database modifications. We present a model-based approach to generate stubs for Web Services which respect both an extra-functional contract expressed via a Service Level Agreement (SLA), and a functional contract modeled via a state machine. These stubs allow a developer to set up a testbed over the target platform, in which the extra-functional and functional behavior of a Web Service under development can be tested before its publication.

Deriving test plans from architectural descriptions

The paper presents an approach for deriving test plans for the conformance testing of a system implementation with respect to the formal description of its software architecture (SA). The SA describes a system in terms of its components and connections, therefore the derived test plans address the integration testing phase. We base our approach on a labelled transition system (LTS) modeling the SA dynamics, and on suitable abstractions of it, the Abstract Labelled Transition Systems (ALTSs). ALTSs offer specific views of the SA dynamics by concentrating on relevant features and abstracting away from uninteresting ones. ALTS is a tool we provide to the software architect that lets him/her focus on relevant behavioral patterns and more easily identify those that are meaningful for validation purposes. Intuitively, deriving an adequate set of functional test classes means deriving a set of paths appropriately covering the ALTS. We describe our approach in the scope of a real world case study and discuss in detail all the steps of our methodology, from ALTS identification to test plan generation.