Sebastian Bauersfeld

TESTAR: Tool Support for Test Automation at the User Interface Level

Testing applications with a graphical user interface GUI is an important, though challenging and time consuming task. The state of the art in the industry are still capture and replay tools, which may simplify the recording and execution of input sequences, but do not support the tester in finding fault-sensitive test cases and leads to a huge overhead on maintenance of the test cases when the GUI changes. In earlier works the authors presented the TESTAR tool, an automated approach to testing applications at the GUI level whose objective is to solve part of the maintenance problem by automatically generating test cases based on a structure that is automatically derived from the GUI. In this paper they report on their experiences obtained when transferring TESTAR in three different industrial contexts with decreasing involvement of the TESTAR developers and increasing participation of the companies when deploying and using TESTAR during testing. The studies were successful in that they reached practice impact, research impact and give insight into ways to do innovation transfer and defines a possible strategy for taking automated testing tools into the market.

A metaheuristic approach to test sequence generation for applications with a GUI

As the majority of todays software applications employ a graphical user interface (GUI), it is an important though challenging task to thoroughly test those interfaces. Unfortunately few tools exist to help automating the process of testing. Despite of their well-known deficits, scripting- and capture and replay applications remain among the most common tools in the industry. In this paper we will present an approach where we treat the problem of generating test sequences to GUIs as an optimization problem. We employ ant colony optimization and a relatively new metric called MCT (Maximum Call Tree) to search fault-sensitive test cases. We therefore implemented a test environment for Java SWT applications and will present first results of our experiments with a graphical editor as our main application under test.

Unit Testing Tool Competition

This paper describes the Java Unit Testing Tool Competition that ran in the context of the Search Based Software Testing (SBST) workshop at ICST 2013. It describes the main objective of the benchmark, the Java classes that were selected, the data that was collected, the tools that were used for data collection, the protocol that was carried out to execute the benchmark and how the final benchmark score for each participating tool can be calculated.

GUITest: a Java library for fully automated GUI robustness testing

Graphical User Interfaces (GUIs) are substantial parts of todays applications, no matter whether these run on tablets, smartphones or desktop platforms. Since the GUI is often the only component that humans interact with, it demands for thorough testing to ensure an efficient and satisfactory user experience. Being the glue between almost all of an applications components, GUIs also lend themselves for system level testing. However, GUI testing is inherently difficult and often involves great manual labor, even with modern tools which promise automation. This paper introduces a Java library called GUITest, which allows to generate fully automated GUI robustness tests for complex applications, without the need to manually generate models or input sequences. We will explain how it operates and present first results on its applicability and effectivity during a test involving Microsoft Word.

Unit Testing Tool Competitions - Lessons Learned

This paper reports about the two rounds of the Java Unit Testing Tool Competition that ran in the context of the Search Based Software Testing (SBST) workshop at ICST 2013 and the first Future Internet Testing (FITTEST) workshop at ICTSS 2013. It describes the main objectives of the benchmark, the Java classes that were selected in both competitions, the data that was collected, the tools that were used for data collection, the protocol that was carried out to execute the benchmark and how the final benchmark scores for each participating tool were calculated. Eventually, we discuss the challenges encountered during the events, what we learned and how we plan to improve our framework for future competitions.

Evaluating the TESTAR tool in an industrial case study

[Context] Automated test case design and execution at the GUI level of applications is not a fact in industrial practice. Tests are still mainly designed and executed manually. In previous work we have described TESTAR, a tool which allows to set-up fully automatic testing at the GUI level of applications to find severe faults such as crashes or non-responsiveness. [Method] This paper aims at the evaluation of TESTAR with an industrial case study. The case study was conducted at SOFTEAM, a French software company, while testing their Modelio SaaS system, a cloud-based system to manage virtual machines that run their popular graphical UML editor Modelio. [Goal] The goal of the study was to evaluate how the tool would perform within the context of SOFTEAM and on their software application. On the other hand, we were interested to see how easy or difficult it is to learn and implant our academic prototype within an industrial setting. [Results] The effectiveness and efficiency of the automated tests generated with TESTAR can definitely compete with that of the manual test suite. [Conclusions] The training materials as well as the user and installation manual of TESTAR need to be improved using the feedback received during the study. Finally, the need to program Java-code to create sophisticated oracles for testing created some initial problems and some resistance. However, it became clear that this could be solved by explaining the need for these oracles and compare them to the alternative of more expensive and complex human oracles. The need to raise consciousness that automated testing means programming solved most of the initial problems.

Evaluating rogue user testing in industry: An experience report

Testing applications with a graphical user interface (GUI) is an important, though challenging and time consuming task. The state of the art in the industry are still capture and replay tools, which may simplify the recording and execution of input sequences, but do not support the tester in finding fault-sensitive test cases and leads to a huge overhead on maintenance of the test cases when the GUI changes. In earlier works we presented the Rogue User Testing Tool, an automated approach to testing applications at the GUI level whose objective is to solve part of the maintenance problem by automatically generating test cases based on a structure that is automatically derived from the GUI. In this paper we report on our experiences obtained when implanting the Rogue User testing Tool with the Spanish software vendor Clavei who decided to apply the tool to stress test a component of one of their ERP applications. Our main goal was to identify potential problems that arise during the setup of the Rogue User. While carrying out our tests, we discovered critical and previously unknown faults in the application under test.

An approach to automatic input sequence generation for GUI testing using ant colony optimization

Testing applications with a graphical user interface (GUI) is an important, though challenging and time consuming task. The state of the art in the industry are still capture and replay tools, which greatly simplify the recording and execution of input sequences, but do not support the tester in finding fault-sensitive test cases. While search-based test case generation strategies, such as evolutionary testing, are well researched for various areas of testing, relatively little work has been done on applying these techniques to an entire GUI of an application. This paper presents an approach to finding input sequences for GUIs using ant colony optimization and a relatively new metric called maximum call stacks for use within the fitness function.

Future Internet Testing

Future Internet applications are expected to be much more complex and powerful, by exploiting various dynamic capabilities For testing, this is very challenging, as it means that the range of possible behavior to test is much larger, and moreover it may at the run time change quite frequently and significantly with respect to the assumed behavior tested prior to the release of such an application. The traditional way of testing will not be able to keep up with such dynamics. The Future Internet Testing (FITTEST) project (http://crest.cs.ucl.ac.uk/ fittest/), a research project funded by the European Commission (grant agreement n. 257574) from 2010 till 2013, was set to explore new testing techniques that will improve our capacity to deal with the challenges of testing Future Internet applications. Such techniques should not be seen as replacement of the traditional testing, but rather as a way to complement it. This paper gives an overview of the set of tools produced by the FITTEST project, implementing those techniques.

Automatische Generierung von Testsequenzen für GUI-basierte Anwendungen mit Ant Colony Optimization

Die Automatisierung des Tests von Anwendungen mit grafischer Benutzerschnittstelle (graphical user interface; GUI) ist trotz einer grosen Anzahl von verfugbaren Testwerkzeugen in der Praxis immer noch unbefriedigend gelost. GUI-Tests werden meist mit Capture-Replay-Werkzeugen aufgezeichnet und wieder abgespielt oder mit Hilfe von Testskripten implementiert. Bei Anderungen am grafischen Design der Benutzerschnittstelle lassen sich die aufgezeichneten oder implementierten Testskripte nicht direkt weiterverwenden, sondern mussen je nach Umfang der Anderungen am GUI mehr oder weniger stark angepasst werden. Haufig leidet darunter die Akzeptanz der Werkzeuge, so dass auch manuelle Tests weit verbreitet sind [1]. Ziel unserer Arbeiten ist es, den Test von GUI-basierten Anwendungen von der Testfallermittlung bis zur Testausfuhrung zu automatisieren. Dabei sollen Tests generiert werden, welche die Funktionen des Pruflings moglichst umfassend uberdecken. Die automatische Generierung von Interaktionsfolgen soll einerseits eine deutliche Erhohung der Testtiefe und damit eine Steigerung der Testqualitat ermoglichen und andererseits fur eine Verbesserung der Testeffizienz und damit fur eine Kostenreduzierung der Tests sorgen.