Mauro Santoro

AutoBlackTest: Automatic Black-Box Testing of Interactive Applications

Automatic test case generation is a key ingredient of an efficient and cost-effective software verification process. In this paper we focus on testing applications that interact with the users through a GUI, and present AutoBlackTest, a technique to automatically generate test cases at the system level. AutoBlackTest uses reinforcement learning, in particular Q-Learning, to learn how to interact with the application under test and stimulate its functionalities. The empirical results show that AutoBlackTest can execute a relevant portion of the code of the application under test, and can reveal previously unknown problems by working at the system level and interacting only through the GUI.

Learning extended FSA from software: An empirical assessment

A number of techniques that infer finite state automata from execution traces have been used to support test and analysis activities. Some of these techniques can produce automata that integrate information about the data-flow, that is, they also represent how data values affect the operations executed by programs. The integration of information about operation sequences and data values into a unique model is indeed conceptually useful to accurately represent the behavior of a program. However, it is still unclear whether handling heterogeneous types of information, such as operation sequences and data values, necessarily produces higher quality models or not. In this paper, we present an empirical comparative study between techniques that infer simple automata and techniques that infer automata extended with information about data-flow. We investigate the effectiveness of these techniques when applied to traces with different levels of sparseness, produced by different software systems. To the best of our knowledge this is the first work that quantifies both the effect of adding data-flow information within automata and the effectiveness of the techniques when varying sparseness of traces.

AutoBlackTest: a tool for automatic black-box testing

In this paper we present AutoBlackTest, a tool for the automatic generation of test cases for interactive applications. AutoBlackTest interacts with the application though its GUI, and uses reinforcement learning techniques to understand the interaction modalities and to generate relevant testing scenarios. Early results show that the tool has the potential of automatically discovering bugs and generating useful system and regression test suites.

Link: exploiting the web of data to generate test inputs

Applications that process complex data, such as maps, personal data, book information, travel data, etc., are becoming extremely common. Testing such applications is hard, because they require realistic and coherent test inputs that are expensive to generate manually and difficult to synthesize automatically. So far the research on test case generation techniques has focused mostly on generating test sequences and synthetic test inputs, and has payed little attention to the generation of complex test inputs. This paper presents Link, a technique to automatically generate test cases for applications that process complex data. The novel idea of Link is to exploit the Web of Data to generate test data that match the semantics of the related fields, and satisfy the semantic constraints that arise among interrelated fields. Link automatically analyzes the GUI of the application under test, generates a model of the required inputs, queries DBPedia to extract the data that can be used in the tests, and uses the extracted data to generate complex system test inputs. The experimental results show that Link can generate realistic and coherent test inputs that can exercise behaviors difficult to exercise with currently available techniques.

Automatic testing of GUI-based applications

Testing GUI‐based applications is hard and time consuming because it requires exploring a potentially huge execution space by interacting with the graphical interface of the applications. Manual testing can cover only a small subset of the functionality provided by applications with complex interfaces, and thus, automatic techniques are necessary to extensively validate GUI‐based systems. This paper presents AutoBlackTest, a technique to automatically generate test cases at the system level. AutoBlackTest uses reinforcement learning, in particular Q‐learning, to learn how to interact with the application under test and stimulate its functionalities. When used to complement the activity of test designers, AutoBlackTest reuses the information in the available test suites to increase its effectiveness. The empirical results show that AutoBlackTest can sample better than state of the art techniques the behaviour of the application under test and can reveal previously unknown problems by working at the system level and interacting only through the graphical user interface. Copyright

SEIM: static extraction of interaction models

The quality of systems that integrate Web services provided by independent organizations depends on the ways the systems interact with the services, i.e., on their interaction protocols, which are not always easy to deduce by inspecting the code. Accurate models of interaction protocols provide a comprehensive view of the interactions, and support manual and automatic analysis of corner cases that are often difficult to discover, and are responsible from many subtle failures. Models of program behavior can be extracted either statically from the source code or dynamically from execution traces. Dynamic techniques cannot reveal behaviors that have not been executed, and thus dynamic models are partially useful to identify critical corner cases. Static techniques produce models of the whole execution space, and thus static models are more useful to find critical cases. Most static techniques to extract models of the code behavior focus on the usage protocol, i.e., how the services can be used, rather than the interaction protocol, i.e., how applications use services. While usage protocols can help software engineers understanding the service behavior, interaction protocols describe the subset of actual interactions between the system and the integrated services, and provide the information required to understand critical interactions. In this paper, we present SEIM, a technique to extract models of the service interaction protocol. SEIM extracts models statically, and includes a novel refinement strategy to eliminate infeasible behaviors that reduce the usability of statically derived models. This paper describes the technique and provides some empirical data from the early experience of SEIM with an application that interacts with the eBay Web Services.

Extracting widget descriptions from GUIs

Graphical User Interfaces (GUIs) are typically designed to simplify data entering, data processing and visualization of results. However, GUIs can also be exploited for other purposes. For instance, automatic tools can analyze GUIs to retrieve information about the data that can be processed by an application. This information can serve many purposes such as ease application integration, augment test case generation, and support reverse engineering techniques. In the last years, the scientific community provided an increasing attention to the automatic extraction of information from interfaces. For instance, in the domain of Web applications, learning techniques have been used to extract information from Web forms. The knowledge about the data that can be processed by an application is not only relevant for the Web, but it is also extremely useful to support the same techniques when applied to desktop applications. In this paper we present a technique for the automatic extraction of descriptive information about the data that can be handled by widgets in GUI-based desktop applications. The technique is grounded on mature standards and best practices about the design of GUIs, and exploits the presence of textual descriptions in the GUIs to automatically obtain descriptive data for data widgets. The early empirical results with three desktop applications show that the presented algorithm can extract data with high precision and recall, and can be used to improve generation of GUI test cases.

Detecting precise behavioral models

Models of the dynamic behavior of program executions can support testing and analysis activities. Deriving complete and precise models is hard, and model imprecision and incompleteness affect the effectiveness of model-based testing and analysis. In my PhD thesis, I aim to develop a framework to generate precise behavioral models. The approach combines dynamic analysis, static analysis and hybrid analysis. In this paper, I describe the core elements of the approach that should be applicable to multiple types of models and systems.

G-RankTest: Dynamic Analysis and Testing of Upgrades in LabVIEW Software

In this chapter we present G-RankTest, a technique for the automatic generation, ranking, and execution of regression test cases for controller applications.

G-RankTest: regression testing of controller applications

Since controller applications must typically satisfy real-time constraints while manipulating real-world variables, their implementation often results in programs that run extremely fast and manipulate numerical inputs and outputs. These characteristics make them particularly suitable for test case generation. In fact a number of test cases can be easily created, due to the simplicity of numerical inputs, and executed, due to the speed of computations. In this paper we present G-RankTest, a technique for test case generation and prioritization. The key idea is that test case generation can run for long sessions (e.g., days) to accurately sample the behavior of a controller application and then the generated test cases can be prioritized according to different strategies, and used for regression testing every time the application is modified. In this work we investigate the feasibility of using the gradient of the output as a criterion for selecting the test cases that activate the most tricky behaviors, which we expect easier to break when a change occurs, and thus deserve priority in regression testing.