André Takeshi Endo

Model-Based Testing of Service-Oriented Applications via State Models

Service-oriented architectures and web services have been used to foster the development of loosely coupled, interoperable, and distributed applications. Mission-critical and business process systems can be implemented with them, requiring a high level of quality. Model-based testing allied with state models is a promising candidate due to its efficiency, effectiveness, and flexibility. In this paper, we propose a model-based testing process to verify service-oriented applications. Finite state machines are used to model and support the test case generation. We evaluated the applicability of our process with a case study using a prototype tool.

Event- and Coverage-Based Testing of Web Services

Service-Oriented Architecture (SOA) fosters the development of loosely coupled applications. Web services have been favored as a promising technology to implement SOAs. Since web services are often involved in complex business processes and safety-critical systems, it is important that they are of high level of reliability. In this paper a strategy for event-based testing of web services is introduced which enables a systematic test case generation for structural testing with the focus on code coverage. The novelty of the approach stems from its integration of an event-based view with a coverage-based view. This forms a combination of black- and white-box testing leading to grey-box testing. A bank service example illustrates the strategy.

Quantifying the Characteristics of Java Programs That May Influence Symbolic Execution from a Test Data Generation Perspective

Testing plays a key role in assessing the quality of a software product. During testing, a program is run in hopes of finding faults. As exhaustive testing is seldom possible, specific testing criteria have been proposed to help testers to devise test cases that cover the most relevant faulty scenarios. Manually creating test cases that satisfy these criteria is time consuming, error prone, and unwieldy. Symbolic execution has been used as an effective way of automatically generating test data that meets those criteria. Although this technique has been used for over three decades, several challenges remain, such as path explosion, precision of floating-point data, constraints with complex expressions, and dependency of external libraries. In this paper, we explore a sample of 100 open source Java projects in order to analyze characteristics that are relevant to generate test data using symbolic execution. The results provide valuable insight into how researchers and practitioners can tailor symbolic execution techniques and tools to better suit the needs of different Java applications.

Experimental Comparison of Test Case Generation Methods for Finite State Machines

Testing from finite state machines has been widely investigated due to its well-founded and sound theory as well as its practical application in different areas, e.g., Web-based systems and protocol testing. There has been a recurrent interest in developing methods capable of generating test suites that detect all faults in a given fault domain. However, the proposal of new methods motivates the comparison with traditional methods. In this context, we conducted a set of experiments that compares W, HSI, H, SPY, and P methods. The results have shown that H, SPY, and P methods produce smaller test suites than traditional methods (W, HSI). Although the P method presented the shortest test suite in most cases, its reduction is smaller compared with H and SPY. We have also observed that the reduction ratio in partial machines is smaller than that in complete machines.

An empirical study to quantify the characteristics of Java programs that may influence symbolic execution from a unit testing perspective

We analyze two benchmarks: SF100 (100 projects) and R47 (47 projects).We show the distributions of issues on generating test data using symbolic execution.We show that few methods have potential for path explosion.We show that dependency and exception-dependent paths are very frequent.We show that the constraints complexity is mainly affected by object types. In software testing, a program is executed in hopes of revealing faults. Over the years, specific testing criteria have been proposed to help testers to devise test cases that cover the most relevant faulty scenarios. Symbolic execution has been used as an effective way of automatically generating test data that meet those criteria. Although this technique has been used for over three decades, several challenges remain and there is a lack of research on how often they appear in real-world applications. In this paper, we analyzed two samples of open source Java projects in order to understand the characteristics that may hinder the generation of unit test data using symbolic execution. The first sample, named SF100, is a third party corpus of classes obtained from 100 projects hosted by SourceForge. The second sample, called R47, is a set of 47 well-known and mature projects we selected from different repositories. Both samples are compared with respect to four dimensions that influence symbolic execution: path explosion, constraint complexity, dependency, and exception-dependent paths. The results provide valuable insight into how researchers and practitioners can tailor symbolic execution techniques and tools to better suit the needs of different Java applications.

Event tree algorithms to generate test sequences for composite Web services

In service‐oriented architecture testing, the behaviour of a composite service and its partner services can be represented as an event graph (model). As test cases are derived from such a model, the Chinese postman problem (CPP) algorithm is applied to cover all edges. Although CPP provides an optimal solution, there is a lack of alternative algorithms to obtain not only a faster execution but also a satisfactory solution. This paper introduces the event tree algorithm (ETA) to generate test cases from event sequence models for Web service compositions. Two strategies are also proposed to improve ETA: First, a greedy strategy to build the tree is defined based on the cost of expanding given edges. Second, redundant elements are eliminated (pruned) in the tree. Four versions of ETA are compared with CPP with respect to test suite size, generation time, number of test sequences, and test sequence length.

An analysis of automated tests for mobile Android applications

Mobile computing has become ubiquitous, thus the amount and complexity of mobile applications have challenged software engineering practices. To overcome the challenges brought by mobile applications, the adoption of repeatable, systematic and mainly automated tests have been researched. In this paper, we look into open source projects in hopes of identifying how automated tests are applied to mobile applications developed for the Android platform. We analyzed the automated tests to identify the frameworks adopted, the relation between the production and the testing code, and how the following challenges have been dealt with: connectivity, rich GUIs, limited resources, sensors, and multiple configurations.

Fault domain-based testing in imperfect situations: a heuristic approach and case studies

Model-based testing (MBT) involves creating an abstraction, called a model, to represent the system and automatically deriving test cases from this model. MBT can be performed using various approaches that generally employ certain assumptions or requirements affecting the test performance in practice. Here, we consider the harmonized state identifiers (HSI) method, which is based on finite state machine (FSM) models and generates test sets that cover all faults in a given domain under certain conditions. We are interested in the application of the HSI method in practical scenarios where some conditions do not hold or are not straightforward to satisfy. Thus, we propose a heuristic extension to the HSI method, called heuristic HSI (HHSI), to consider imperfect situations as they often occur in practice. To analyze the characteristics of HHSI, we empirically compare it to random testing and coverage-based testing using non-trivial case studies. The experiments include model-based mutation analyses over several FSM models.

Generation of Checking Sequences Using Identification Sets

Finite state machine-based testing aims at generating checking sequences that guarantee the conformance between the implementation and the specification of a system. For that purpose, several methods have been proposed to generate checking sequences which ensure full coverage of possible faults in the implementation. Many existing methods are based on a special sequence, called distinguishing sequence, which does not exist for every minimal machine. Some methods are based on characterization sets since they exist for every machine. However, these methods generate checking sequences exponentially long. In this paper, we propose a method to generate checking sequences using identification sets. These sets exist for every minimal FSM and also lead to shorter checking sequences. We conducted an experimental study to compare the proposed method with the existing methods. The results show that on average our method generates checking sequences 31.7% to 99.9% shorter than the ones produced by existing methods.

Characterizing mobile apps from a source and test code viewpoint

Abstract Context: while the mobile computing market has expanded and become critical, the amount and complexity of mobile apps have also increased. To assure reliability, these apps require software engineering methods, mainly verification, validation, and testing. However, mobile app testing is a challenging activity due to the diversity and limitations found in mobile devices. Thus, it would be interesting to characterize mobile apps in hopes of assisting in the definition of more efficient and effective testing approaches. Objective: this paper aims to identify and quantify the specific characteristics of mobile apps so that testers can draw from this knowledge and tailor software testing activities to mobile apps. We investigate the presence of automated tests, adopted frameworks, external connectivity, graphical user interface (GUI) elements, sensors, and different system configurations. Method: we developed a tool to support the automatic extraction of characteristics from Android apps. We conducted an empirical study with a sample of 663 open source mobile apps. Results: we found that one third of the projects perform automated testing. The frameworks used in these projects can be divided into three groups: unit testing, GUI testing, and mocking. There is a medium correlation between project size and test presence. Specific features of mobile apps (connectivity, GUI, sensors, and multiple configurations) are present in the projects, however, they are fully covered by tests. Conclusion: automated tests are still not developed in a systematic way. Interestingly, measures of app popularity (number of downloads and rating) do not seem to be correlated with the presence of tests. However, the results show a correlation of the project size and more critical domains with the existence of automated tests. Although challenges such as connectivity, sensors, and multiple configurations are present in the examined apps, only one tool has been identified to support the testing of these challenges.