Valdivino Alexandre de Santiago Júnior

Generating model-based test cases from natural language requirements for space application software

Natural Language (NL) deliverables suffer from ambiguity, poor understandability, incompleteness, and inconsistency. Howewer, NL is straightforward and stakeholders are familiar with it to produce their software requirements documents. This paper presents a methodology, SOLIMVA, which aims at model-based test case generation considering NL requirements deliverables. The methodology is supported by a tool that makes it possible to automatically translate NL requirements into Statechart models. Once the Statecharts are derived, another tool, GTSC, is used to generate the test cases. SOLIMVA uses combinatorial designs to identify scenarios for system and acceptance testing, and it requires that a test designer defines the application domain by means of a dictionary. Within the dictionary there is a Semantic Translation Model in which, among other features, a word sense disambiguation method helps in the translation process. Using as a case study a space application software product, we compared SOLIMVA with a previous manual approach developed by an expert under two aspects: test objectives coverage and characteristics of the Executable Test Cases. In the first aspect, the SOLIMVA methodology not only covered the test objectives associated to the expert’s scenarios but also proposed a better strategy with test objectives clearly separated according to the directives of combinatorial designs. The Executable Test Cases derived in accordance with the SOLIMVA methodology not only possessed similar characteristics with the expert’s Executable Test Cases but also predicted behaviors that did not exist in the expert’s strategy. The key benefits from applying the SOLIMVA methodology/tool within a Verification and Validation process are the ease of use and, at the same time, the support of a formal method consequently leading to a potential acceptance of the methodology in complex software projects.

A Controlled Experiment for Combinatorial Testing

In this paper, we present a controlled experiment for combinatorial designs algorithms aiming at software test case generation. We compare our recently proposed algorithm, TTR, to generate Mixed-Level Covering Array (MCA) with four other well-known combinatorial designs algorithms/tools regarding two aspects: cost in terms of the size of the set of test cases, and cost in terms of the time to generate the test suites. We used a set of 27 instances for this experiment. Results show that our algorithm was the best in terms of the size of the test suite, but was the poorest in terms of the time to generate the test cases. However, the not so good performance of our algorithm regarding the time to generate the test suite can be alleviated by the fact that TTR produces shorter set of test cases to be executed. We also made a comparison about the similarity of the test cases, i.e. to realize how similar are the test suites (test input data) of TTR compared with the other four algorithms/tools. We conclude that the TTRs test suite is not similar to any other test suites meaning that our algorithm has the potential to uncover different software defects by exercising different parts of the Software Under Test (SUT).

T-Tuple Reallocation: An Algorithm to Create Mixed-Level Covering Arrays to Support Software Test Case Generation

A fact that is known both by researchers and by industry professionals is that exhaustive software testing is impractical. Therefore, one of the most studied activities of software testing process is the generation/selection of test cases. However, selecting test cases that reveal the greatest number of defects within a software is a challenging task, due to the significantly high amount of entries that the system can receive, and even due to the different characteristics of software products in several application domains. This work presents a new algorithm, called T-Tuple Reallocation TTR, to generate Mixed-Level Covering Array MCA which is one of the techniques of combinatorial designs that aims at test case generation. After studying various algorithms/techniques to generate combinatorial designs, starting with pairwise design, TTR was proposed aiming at decreasing the amount of test cases produced to test a software product. The new algorithm was able to create shorter sets of test cases in comparison with classical algorithms/tools proposed in the literature. Although TTR, in general, demanded longer time to generate the sets of test cases, this rise in time can be compensated by a smaller number of test cases so that less time is required for executing them. In the end, this may imply less time for accomplishing the testing process as a whole.

WEB-PerformCharts: a collaborative web-based tool for test case generation from statecharts

Distributed development of software has turned into a natural and modern approach where teams spread over the world cooperate to develop a software product, and this has become possible due to the expansion and popularity of global networks as internet. Collaborative tools coordinate a variety of tasks of several members of a team with an objective of reaching a specific goal. One such task that plays a major role, within the software development life cycle, is testing. In particular this task becomes more and more important when considering critical software such as space applications, which is the case of Brazilian Space Institutions CTA and INPE. The work discussed in this paper has two objectives: (i) present a web-based tool, WEB-PerformCharts, that can generate black-box test cases of a space application software; (ii) show that Statecharts are an excellent option to model the software specification, from which test sequences can be generated by applying several methods well known from the published literature.

From Statecharts into Model Checking: A Hierarchy-based Translation and Specification Patterns Properties to Generate Test Cases

Complexity and notation of formal methods are still major impediments for a wider use of these mathematical-based approaches in Software Engineering which include its adoption in software testing. While formal, Statecharts are relatively simple to use and many projects in different domains have been relying on them. In this paper, we present a hierarchy-based translation method, HiMoST, to generate software test cases via Model Checking. Starting with a behavioral modeling in Harels Statecharts, we propose a method to translate from Statecharts into a general structure based on the NuSMV language, and we formalize CTL properties by means of specification patterns and a Combinatorial Interaction Testing algorithm. We also present a cost-effectiveness evaluation (quasiexperiment) to compare four different patterns/pattern scopes. Results indicate that the Precedence Chain (P precedes S, T) pattern with Global scope presents the best performance.Complexity and notation of formal methods are still major impediments for a wider use of these mathematical-based approaches in Software Engineering which include its adoption in software testing. While formal, Statecharts are relatively simple to use and many projects in different domains have been relying on them. In this paper, we present a hierarchy-based translation method, HiMoST, to generate software test cases via Model Checking. Starting with a behavioral modeling in Harels Statecharts, we propose a method to translate from Statecharts into a general structure based on the NuSMV language, and we formalize CTL properties by means of specification patterns and a Combinatorial Interaction Testing algorithm. We also present a cost-effectiveness evaluation (quasiexperiment) to compare four different patterns/pattern scopes. Results indicate that the Precedence Chain (P precedes S, T) pattern with Global scope presents the best performance.

Time Performance Formal Evaluation of Complex Systems

Formal verification methods, such as Model Checking, have been used for addressing performance/dependability analysis of systems. Such formal methods have several advantages over traditional techniques aiming at performance/dependability analysis such as the use of a single computational technique for evaluation of any measure and all complex numerical computation steps are hidden to the user. This paper reports on the use of Probabilistic Model Checking for time performance evaluation of complex systems. We propose an approach, TPerP, that allows a professional to clearly address time performance analysis based on Continuous-Time Markov Chain (CTMC). Our approach takes into consideration several types of delay analyzes. We applied it to a balloon-borne high energy astrophysics scientific experiment where we dealt with CTMCs that had around 30 million reachable states and 75 million transitions, and we concluded that the current definition used in the balloon system is inadequate in terms of performance.

Tool support for generating model-based test cases via web

Testing activities play an important role in order to obtain high quality software products. These activities become more important when considering critical software, for instance, space application software. Nowadays, there is an extensive collaboration among space institutions. So, it is more than natural to expect distributed development of software and software testing activities. Therefore, a collaborative tool hosted on the internet becomes quite useful. In this respect, WEB-PerformCharts 2.0 tool discussed in this paper moves in this direction. The tool focuses on supporting a single aspect of distributed software development: the activity of generating test cases via web. Moreover, it allows model-based test case generation by means of formal methods formal languages statecharts and FSM which are considered state of the art in software development. WEB-PerformCharts 2.0 can be used to generate test cases for any kind of reactive systems modelled in statecharts or FSM. We present three case studies in different application domains to demonstrate the feasibility of our tool.

Transformation of UML Behavioral Diagrams to Support Software Model Checking.

Unified Modeling Language (UML) is currently accepted as the standard for modeling (object-oriented) software, and its use is increasing in the aerospace industry. Verification and Validation of complex software developed according to UML is not trivial due to complexity of the software itself, and the several different UML models/diagrams that can be used to model behavior and structure of the software. This paper presents an approach to transform up to three different UML behavioral diagrams (sequence, behavioral state machines, and activity) into a single Transition System to support Model Checking of software developed in accordance with UML. In our approach, properties are formalized based on use case descriptions. The transformation is done for the NuSMV model checker, but we see the possibility in using other model checkers, such as SPIN. The main contribution of our work is the transformation of a non-formal language (UML) to a formal language (language of the NuSMV model checker) towards a greater adoption in practice of formal methods in software development.

H-Switch Cover: a new test criterion to generate test case from finite state machines

Test cases generation based on Finite State Machines (FSMs) has been addressed for quite some time. Model-based testing has drawn attention from researchers and practitioners as one of the approaches to support software verification and validation. Several test criteria have been proposed in the literature to generate test cases based on formal methods, such as FSM. However, there is still a lot to be done on this aspect in order to clearly direct a test designer to choose a test criterion most suitable to generate test cases for a certain application domain. This work presents a new test criterion for model-based test case generation based on FSM, H-Switch Cover. H-Switch Cover relies on the traditional Switch Cover test criterion, but H-Switch Cover uses new heuristics to improve its performance, for example, adoption of rules to optimize graph balancing and traverse the graph for test cases generation. We conducted an investigation of cost and efficiency of this new test criterion by comparing it with unique input/output and distinguishing sequence. We used two embedded software products (space application software products) and mutation analysis for assessing efficiency. In general, for the case studies proposed in this paper in terms of cost (amount of events) and efficiency (mutation score), H-Switch Cover test criterion presented an average and a standard deviation better than the other two test criteria.

An algorithm for combinatorial interaction testing: definitions and rigorous evaluations

BackgroundCombinatorial Interaction Testing (CIT) approaches have drawn attention of the software testing community to generate sets of smaller, efficient, and effective test cases where they have been successful in detecting faults due to the interaction of several input parameters. Recent empirical studies show that greedy algorithms are still competitive for CIT. It is thus interesting to investigate new approaches to address CIT test case generation via greedy solutions and to perform rigorous evaluations within the greedy context.MethodsWe present a new greedy algorithm for unconstrained CIT, T-TupleReallocation (TTR), to generate CIT test suites specifically via the Mixed-value Covering Array (MCA) technique. The main reasoning behind TTR is to generate an MCA M by creating and reallocating t-tuples into this matrix M, considering a variable called goal (ζ). We performed two controlled experiments addressing cost-efficiency and only cost. Considering both experiments, we did 3200 executions related to 8 solutions. In the first controlled experiment, we compared versions 1.1 and 1.2 of TTR in order to check whether there is significant difference between both versions of our algorithm. In such experiment, we jointly considered cost (size of test suites) and efficiency (time to generate the test suites) in a multi-objective perspective. In the second controlled experiment we confronted TTR 1.2 with five other greedy algorithms/tools for unconstrained CIT: IPOG-F, jenny, IPO-TConfig, PICT, and ACTS. We performed two different evaluations within this second experiment where in the first one we addressed cost-efficiency (multi-objective) and in the second only cost (single objective).ResultsResults of the first controlled experiment indicate that TTR 1.2 is more adequate than TTR 1.1 especially for higher strengths (5, 6). In the second controlled experiment, TTR 1.2 also presents better performance for higher strengths (5, 6) where only in one case it is not superior (in the comparison with IPOG-F). We can explain this better performance of TTR 1.2 due to the fact that it no longer generates, at the beginning, the matrix of t-tuples but rather the algorithm works on a t-tuple by t-tuple creation and reallocation into M.ConclusionConsidering the metrics we defined in this work and based on both controlled experiments, TTR 1.2 is a better option if we need to consider higher strengths (5, 6). For lower strengths, other solutions, like IPOG-F, may be better alternatives.