Diego Clerissi

Capture-replay vs. programmable web testing: An empirical assessment during test case evolution

There are several approaches for automated functional web testing and the choice among them depends on a number of factors, including the tools used for web testing and the costs associated with their adoption. In this paper, we present an empirical cost/benefit analysis of two different categories of automated functional web testing approaches: (1) capture-replay web testing (in particular, using Selenium IDE); and, (2) programmable web testing (using Selenium WebDriver). On a set of six web applications, we evaluated the costs of applying these testing approaches both when developing the initial test suites from scratch and when the test suites are maintained, upon the release of a new software version. Results indicate that, on the one hand, the development of the test suites is more expensive in terms of time required (between 32% and 112%) when the programmable web testing approach is adopted, but on the other hand, test suite maintenance is less expensive when this approach is used (with a saving between 16% and 51%). We found that, in the majority of the cases, after a small number of releases (from one to three), the cumulative cost of programmable web testing becomes lower than the cost involved with capture-replay web testing and the cost saving gets amplified over the successive releases.

Visual vs. DOM-Based Web Locators: An Empirical Study

Automation in Web testing has been successfully supported by DOM-based tools that allow testers to program the interactions of their test cases with the Web application under test. More recently a new generation of visual tools has been proposed where a test case interacts with the Web application by recognising the images of the widgets that can be actioned upon and by asserting the expected visual appearance of the result.

Improving Test Suites Maintainability with the Page Object Pattern: An Industrial Case Study

The page object pattern is used in the context of web testing for abstracting the applications web pages in order to reduce the coupling between test cases and application under test. This paper reports on an industrial case study in a small Italian company (eXact learning solutions S.p.A.) investigating the potential benefits of adopting the page object pattern to improve the maintainability of Selenium WebDriver test cases. After a maintenance/evolution activity performed on the application under test, we compared two equivalent test suites, one built using the page object pattern and one without it. The results of our case study indicate a strong reduction in terms of time required (by a factor of about three) and number of modified LOCs (by a factor of about eight) to repair the test suite when the page object pattern is used.

Approaches and Tools for Automated End-to-End Web Testing

The importance of test automation in web engineering comes from the widespread use of web applications and the associated demand for code quality. Test automation is considered crucial for delivering the quality levels expected by users, since it can save a lot of time in testing and it helps developers to release web applications with fewer defects. The main advantage of test automation comes from fast, unattended execution of a set of tests after some changes have been made to a web application. Moreover, modern web applications adopt a multitier architecture where the implementation is scattered across different layers and run on different machines. For this reason, end-to-end testing techniques are required to test the overall behavior of web applications. In the last years, several approaches have been proposed for automated end-to-end web testing and the choice among them depends on a number of factors, including the tools used for web testing and the costs associated with their adoption. They can be classified using two main criteria: the first concerns how test cases are developed (ie, Capture-Replay and Programmable approaches), while, the second concerns how test cases localize the web elements to interact with (ie, Coordinates-based, DOM-based, and Visual approaches), that is what kind of locators are used for selecting the target GUI components. For developers and project managers it is not easy to select the most suitable automated end-to-end web testing approach for their needs among the existing ones. This chapter provides a comprehensive overview of the automated end-to-end web testing approaches and summarizes the findings of a long term research project aimed at empirically investigating their strengths and weaknesses.

Comparing the maintainability of selenium WebDriver test suites employing different locators: a case study

Test suite maintenance tends to have the biggest impact on the overall cost of test automation. Frequently modifications applied on a web application lead to have one or more test cases broken and repairing the test suite is a time-consuming and expensive task. This paper reports on an industrial case study conducted in a small Italian company investigating on the analysis of the effort to repair web test suites implemented using different UI locators (e.g., Identifiers and XPath). The results of our case study indicate that ID locators used in conjunction with LinkText is the best solution among the considered ones in terms of time required (and LOCs to modify) to repair the test suite to the new release of the application.

What Are the Used UML Diagram Constructs? A Document and Tool Analysis Study Covering Activity and Use Case Diagrams

UML offers a very large set of constructs for each of its diagram types, however many of them seem scarcely used or even their existence is not known. Here, we decided to present a precise view of the usage levels of the constructs of activity and use case diagrams by means of a document and tool analysis study, covering preliminarily: books, courses, tutorials, and tools about UML. Results of the study show that, among the 47 activity diagrams constructs, a large majority of them seem to be scarcely used, while, only nine result widely used, whereas only two of the nine constructs of the use case diagrams seem scarcely used. This work is part of a larger project aimed at investigating the usage level of the UML diagrams and their constructs, also by means of a personal opinion survey intended for UML users.

Service-oriented domain and business process modelling

We present Precise Service-Oriented Modelling (Precise SOM) - a novel lightweight method for integrated domain and business process modelling, which follows the service-oriented paradigm, and uses a UML profile as notation - and a detailed workflow to guide the production of the models. In our method the produced UML models are precisely defined by means of a metamodel, a set of constraints, and a limited set of UML constructs to help modellers to avoid common mistakes and to guarantee, by construction, a good quality. Precise SOM has been successfully used in an industry-academic project concerning the modelling of a big harbour.

An acceptance testing approach for Internet of Things systems

Internet of things (IoT) systems are becoming ubiquitous and assuring their quality is fundamental. Unfortunately, a few proposals for testing these complex, and often safety-critical, systems are present in the literature. The authors propose an approach for acceptance testing of IoT systems adopting graphical user interfaces as a principal way of interaction. Acceptance testing is a type of black box testing based on test scenarios, i.e. sequences of steps/actions performed by the user or the system. In their approach, test scenarios are derived from a state machine that expresses the behaviour of the system under test, and test cases are derived from them by specifying the actual data and assertions and made executable by implementing the corresponding test scripts. As a case study, they selected a mobile health IoT system for diabetes management composed of local sensors/actuators, smartphones, and a remote cloud-based system. The effectiveness of the approach has been evaluated by measuring the capability of two test suites implemented using different localisation strategies (visual and structure-based) in detecting mutants of the original m-health system. Results show the effectiveness of the test suites implemented by following the proposed approach since 93% of the generated mutants have been detected.

Web testware evolution

Web applications evolve at a very fast rate, to accommodate new functionalities, presentation styles and interaction modes. The test artefacts developed during web testing must be evolved accordingly. Among the other causes, one critical reason why test cases need maintenance during web evolution is that the locators used to uniquely identify the page elements under test may fail or may behave incorrectly. The robustness of web page locators used in test cases is thus critical to reduce the test maintenance effort. We present an algorithm that generates robust web page locators for the elements under test and we describe the design of an empirical study that we plan to execute to validate such robust locators.

Repairing Selenium Test Cases: An Industrial Case Study about Web Page Element Localization

This poster presents an industrial case study about test automation and test suite maintenance in the context of Web applications. The Web application under test is a Learning Content Management System (eXact learning LCMS). We analysed the costs associated with the realignment of four equivalent Selenium WebDriver test suites, implemented using the page object pattern and different methods to locate web page elements, to a subsequent release of eXact learning LCMS. In our study, the two ID-based test suites required significantly less maintenance effort than the XPath-based ones.