Nicola Amatucci

Considering Context Events in Event-Based Testing of Mobile Applications

A relevant complexity factor in developing and testing mobile apps is given by their sensibility to changes in the context in which they run. As an example, apps running on a smartphone can be influenced by location changes, phone calls, device movements and many other typologies of context events. In this paper, we address the problem of testing a mobile app as an event-driven system by taking into account both context events and GUI events. We present approaches based on the definition of reusable event patterns for the manual and automatic generation of test cases for mobile app testing. One of the proposed testing techniques, based on a systematic and automatic exploration of the behaviour of an Android app, has been implemented and some preliminary case studies on real apps have been carried out in order to explore their effectiveness.

A general framework for comparing automatic testing techniques of Android mobile apps

Abstract As an increasing number of new techniques are developed for quality assurance of Android applications (apps), there is a need to evaluate and empirically compare them. Researchers as well as practitioners will be able to use the results of such comparative studies to answer questions such as, “What technique should I use to test my app?” Unfortunately, there is a severe lack of rigorous empirical studies on this subject. In this paper, for the first time, we present an empirical study comparing all existing fully automatic “online” testing techniques developed for the Android platform. We do so by first reformulating each technique within the context of a general framework. We recognize the commonalities between the techniques to develop the framework. We then use the salient features of each technique to develop parameters of the framework. The result is a general recasting of all existing approaches in a plug-in based formulation, allowing us to vary the parameters to create instances of each technique, and empirically evaluate them on a common set of subjects. Our results show that (1) the proposed general framework abstracts all the common characteristics of online testing techniques proposed in the literature, (2) it can be exploited to design experiments aimed at performing objective comparisons among different online testing approaches and (3) some parameters that we have identified influence the performance of the testing techniques.

Improving code coverage in android apps testing by exploiting patterns and automatic test case generation

This work aims at defining a procedure and a set of mechanisms able to improve the quality of the code coverage in automated software reverse engineering processes, and specifically in automated GUI-driven testing of Android apps. Existing automated model-based testing techniques, based on reverse engineering, generate test cases which can be executed directly on the softwares GUI. We propose to augment the code coverage of these techniques, by exploiting information from patterns, defined at different levels (application design, state-based model, interaction with Android services), and generating additional test cases that may increase the coverage capability of GUI-Ripping based testing technique. The generation of the additional test cases is accomplished by defining an automatable procedure which exploits an existing GUI testing approach and a pattern based approach used in a different context.

AGRippin: a novel search based testing technique for Android applications

Recent studies have shown a remarkable need for testing automation techniques in the context of mobile applications. The main contributions in literature in the field of testing automation regard techniques such as Capture/Replay, Model Based, Model Learning and Random techniques. Unfortunately, only the last two typologies of techniques are applicable if no previous knowledge about the application under testing is available. Random techniques are able to generate effective test suites (in terms of source code coverage) but they need a remarkable effort in terms of machine time and the tests they generate are quite inefficient due to their redundancy. Model Learning techniques generate more efficient test suites but often they do not not reach good levels of coverage. In order to generate test suites that are both effective and efficient, we propose in this paper AGRippin, a novel Search Based Testing technique founded on the combination of genetic and hill climbing techniques. We carried out a case study involving five open source Android applications that has demonstrated how the proposed technique is able to generate test suites that are more effective and efficient than the ones generated by a Model Learning technique.

A Conceptual Framework for the Comparison of Fully Automated GUI Testing Techniques

Fully automated GUI testing techniques play an important role in the modern software development life cycles. These techniques are implemented by algorithms that automatically traverse the GUI by interacting with it, like robots discovering unexplored spaces. These algorithms are able to define and run test cases on the fly, while the application is in execution. Testing adequacy, performance or costs of such techniques may differ on the basis of different factors. In this paper we will propose an approach for comparing fully automated GUI testing techniques in a systematic manner. The approach is based on a generalized parametric algorithm that abstracts the key aspects of these techniques and provides a conceptual framework that can be used to define and compare different testing approaches. To validate the framework, we exploit it to compare the testing adequacy and the GUI models inferred by 9 fully automated testing techniques obtained by varying the configuration of the algorithm. The experiment is performed on a real Android application.

Automated functional testing of mobile applications: a systematic mapping study

Context Testing is a critical and costly activity in the life cycle of a mobile application, due to the growing request of new applications and to the rapid evolution of mobile devices and frameworks. Testing automation may represent an effective solution to improve the quality of mobile applications and to reduce testing costs. Objective We have performed a systematic mapping study to find, analyze, and classify papers in the scientific literature that are related to the automation of functional testing of mobile applications with the aim to provide a classification scheme useful for researchers and practitioners to have a clear view of the state of the art and to easily find existing solutions to their issues. Method We have conducted the study on the basis of a set of 18 research questions. Search queries have been formulated and applied to 7 search engines and the resulting papers have been filtered by considering sets of inclusion and exclusion criteria. The selected papers have been systematically classified and, in addition, a bibliometric analysis has been performed. Results A systematic map including 131 papers has been obtained and is publicly available. The papers have been classified on the basis of the supported testing activities, the characteristics of the techniques and tools they present, and the evaluation methodologies adopted to validate them. The bibliometric analysis has allowed the identification of the most active researchers, the most attractive venues, and the most influential papers. Conclusions The analysis of the systematic mapping has allowed the identification of some research trends and gaps in this field of study. For example, we have observed a strong prevalence of Android-based approaches, a lack of contributions from industry, and the absence of specific venues and journals focused on mobile testing automation.

Combining Automated GUI Exploration of Android apps with Capture and Replay through Machine Learning

Abstract Context Automated GUI Exploration Techniques have been widely adopted in the context of mobile apps for supporting critical engineering tasks such as reverse engineering, testing, and network traffic signature generation. Although several techniques have been proposed in the literature, most of them fail to guarantee the exploration of relevant parts of the applications when GUIs require to be exercised with particular and complex input event sequences. We refer to these GUIs as Gate GUIs and to the sequences required to effectively exercise them as Unlocking GUI Input Event Sequences. Objective In this paper, we aim at proposing a GUI exploration approach that exploits the human involvement in the automated process to solve the limitations introduced by Gate GUIs, without requiring the preliminary configuration of the technique or the user involvement for the entire duration of the exploration process. Method We propose juGULAR, a Hybrid GUI Exploration Technique combining Automated GUI Exploration with Capture and Replay. Our approach is able to automatically detect Gate GUIs during the app exploration by exploiting a Machine Learning approach and to unlock them by leveraging input event sequences provided by the user. We implement juGULAR in a modular software architecture that targets the Android mobile platform. We evaluate the performance of juGULAR by an experiment involving 14 real Android apps. Results The experiment shows that the hybridization introduced by juGULAR allows to improve the exploration capabilities in terms of Covered Activities, Covered Lines of Code, and generated Network Traffic Bytes at a reasonable manual intervention cost. The experimental results also prove that juGULAR is able to outperform the state-of-the-practice tool Monkey. Conclusion We conclude that the combination of Automated GUI Exploration approaches with Capture and Replay techniques is promising to achieve a thorough app exploration. Machine Learning approaches aid to pragmatically integrate these two techniques.

Towards a Thing-In-the-Loop approach for the Verification and Validation of IoT systems

The Internet of Things (IoT) is rapidly increasing its diffusion, posing great challenges to the research community. IoT systems are composed by smart objects (Things) that are interconnected in order to provide new products and services. The interaction of heterogeneous and distributed smart things guided by software with the physical world brings new sources of safety issues. To this reason, providing valuable and effective solutions to support the verification and validation of such systems is needed. In this paper we introduce a model-driven Thing-In-the-Loop verification and validation approach that transfers the best practices adopted in different embedded system domains towards the IoT world. Starting from models and scenarios representing the structure and behaviors of the IoT system as well as models of its context our approach generates appropriate test cases that are executed in accordance with Model-in-the-Loop, Software-in-the-Loop and Hardware-in-the-Loop techniques. We preliminarily evaluated the feasibility of our approach by applying it in the context of a Smart Mobility case study.