Leonardo Mariani

Automatic generation of software behavioral models

Dynamic analysis of software systems produces behavioral models that are useful for analysis, verification and testing. The main techniques for extracting models of functional behavior generate either models of constraints on data, usually in the form of Boolean expressions, or models of interactions between components, usually in the form of finite state machines. Both data and interaction models are useful for analyzing and verifying different aspects of software behavior, but none of them captures the complex interplay between data values and components interactions. Thus related analysis and testing techniques can miss important information. In this paper, we focus on the generation of models of relations between data values and component interactions, and we present GK-tail, a technique to automatically generate extended finite state machines (EFSMs) from interaction traces. EFSMs model the interplay between data values and component interactions by annotating FSM edges with conditions on data values. We show that EFSMs include details that are not captured by either Boolean expressions or (classic) FSM alone, and allow for more accurate analysis and verification than separate models, even if considered jointly.

Automatic steering of behavioral model inference

Many testing and analysis techniques use finite state models to validate and verify the quality of software systems. Since the specification of such models is complex and time-consuming, researchers defined several techniques to extract finite state models from code and traces. Automatically generating models requires much less effort than designing them, and thus eases the verification and validation of large software systems. However, when models are inferred automatically, the precision of the mining process is critical. Behavioral models mined with imprecise processes can include many spurious behaviors, and can thus compromise the results of testing and analysis techniques that use those models. In this paper, we increase the precision of automata inferred from execution traces, by leveraging two learning techniques. We first mine execution traces to infer statistically significant temporal properties that capture relations between non consecutive and possibly distant events. We then incrementally refine a simple initial automaton by merging likely equivalent states. We identify equivalent states by analyzing set of consecutive events, and we use the inferred temporal properties to evaluate whether two equivalent states can be merged or not. We merge equivalent states only if the merging does violate any temporal property, since a merging that violates temporal properties is likely to introduce an imprecise generalization. Our generalization process that preserves temporal properties while merging states avoids breaking non-local relations, and thus solves one of the major cause of overgeneralized models. Thus, mined properties steer the learning of behavioral models. The technique is completely automated and generates an automaton that both accepts the input traces and satisfies the mined temporal properties. We evaluated our solution by comparing models inferred with and without checking mined temporal properties. Results show that our steering process can significantly improve precision without noticeable loss of recall.

Automated Identification of Failure Causes in System Logs

Log files are commonly inspected by system administrators and developers to detect suspicious behaviors and diagnose failure causes. Since size of log files grows fast, thus making manual analysis impractical, different automatic techniques have been proposed to analyze log files. Unfortunately, accuracy and effectiveness of these techniques are often limited by the unstructured nature of logged messages and the variety of data that can be logged.This paper presents a technique to automatically analyze log files and retrieve important information to identify failure causes. The technique automatically identifies dependencies between events and values in logs corresponding to legal executions, generates models of legal behaviors and compares log files collected during failing executions with the generated models to detect anomalous event sequences that are presented to users. Experimental results show the effectiveness of the technique in supporting developers and testers to identify failure causes.

Dynamic Detection of COTS Component Incompatibility

The development of COTS-based systems shifts the focus of testing and verification from single components to component integration. Independent teams and organizations develop COTS components without referring to specific systems or interaction patterns. Developing systems that reuse COTS components (even high-quality ones) therefore presents new compatibility problems. David Garlan, Robert Allen, and John Ockerbloom (1995) reported that in their experience, integrating four COTS components took 10 person-years (rather than the one planned person-year), mainly because of integration problems. According to Barry Boehm and Chris Abts (1999), three of the four main problems with reusing COTS products are absence of control over their functionality, absence of control over their evolution, and lack of design for interoperability. Our proposed technique, called behavior capture and test, detects COTS component incompatibilities by dynamically analyzing component behavior. BCT incrementally builds behavioral models of components and compares them with the behavior the components display when reused in new contexts. This lets us identify incompatibilities, unexpected interactions, untested behaviors, and dangerous side effects.

Compatibility and Regression Testing of COTS-Component-Based Software

Software engineers frequently update COTS components integrated in component-based systems, and can often chose among many candidates produced by different vendors. This paper tackles both the problem of quickly identifying components that are syntactically compatible with the interface specifications, but badly integrate in target systems, and the problem of automatically generating regression test suites. The technique proposed in this paper to automatically generate compatibility and prioritized test suites is based on behavioral models that represent component interactions, and are automatically generated while executing the original test suites on previous versions of target systems.

Dynamic Analysis for Diagnosing Integration Faults

Many software components are provided with incomplete specifications and little access to the source code. Reusing such gray-box components can result in integration faults that can be difficult to diagnose and locate. In this paper, we present Behavior Capture and Test (BCT), a technique that uses dynamic analysis to automatically identify the causes of failures and locate the related faults. BCT augments dynamic analysis techniques with model-based monitoring. In this way, BCT identifies a structured set of interactions and data values that are likely related to failures (failure causes), and indicates the components and the operations that are likely responsible for failures (fault locations). BCT advances scientific knowledge in several ways. It combines classic dynamic analysis with incremental finite state generation techniques to produce dynamic models that capture complementary aspects of component interactions. It uses an effective technique to filter false positives to reduce the effort of the analysis of the produced data. It defines a strategy to extract information about likely causes of failures by automatically ranking and relating the detected anomalies so that developers can focus their attention on the faults. The effectiveness of BCT depends on the quality of the dynamic models extracted from the program. BCT is particularly effective when the test cases sample the execution space well. In this paper, we present a set of case studies that illustrate the adequacy of BCT to analyze both regression testing failures and rare field failures. The results show that BCT automatically filters out most of the false alarms and provides useful information to understand the causes of failures in 69 percent of the case studies.

CrowdOracles: Can the Crowd Solve the Oracle Problem?

Despite the recent advances in test generation, fully automatic software testing remains a dream: Ultimately, any generated test input depends on a test oracle that determines correctness, and, except for generic properties such as “the program shall not crash”, such oracles require human input in one form or another. CrowdSourcing is a recently popular technique to automate computations that cannot be performed by machines, but only by humans. A problem is split into small chunks, that are then solved by a crowd of users on the Internet. In this paper we investigate whether it is possible to exploit CrowdSourcing to solve the oracle problem: We produce tasks asking users to evaluate CrowdOracles - assertions that reflect the current behavior of the program. If the crowd determines that an assertion does not match the behavior described in the code documentation, then a bug has been found. Our experiments demonstrate that CrowdOracles are a viable solution to automate the oracle problem, yet taming the crowd to get useful results is a difficult task.

AutoBlackTest: Automatic Black-Box Testing of Interactive Applications

Automatic test case generation is a key ingredient of an efficient and cost-effective software verification process. In this paper we focus on testing applications that interact with the users through a GUI, and present AutoBlackTest, a technique to automatically generate test cases at the system level. AutoBlackTest uses reinforcement learning, in particular Q-Learning, to learn how to interact with the application under test and stimulate its functionalities. The empirical results show that AutoBlackTest can execute a relevant portion of the code of the application under test, and can reveal previously unknown problems by working at the system level and interacting only through the GUI.

A Technique for Verifying Component-Based Software

Component-based software systems raise new problems for the testing community: the reuse of components suggests the possibility of reducing testing costs by reusing information about the quality of the software components. This paper addresses the problem of testing evolving software systems, i.e., systems obtained by modifying and/or substituting some of their components. The paper proposes a technique to automatically identify behavioral differences between different versions of the system, to deduce possible problems from inconsistent behaviors. The approach is based on the automatic distilling of invariants from in-field executions. The computed invariants are used to monitor the behavior of new components, and to reveal unexpected interactions. The event generated while monitoring system executions are presented to software engineers who can infer possible problems of the new versions.

Learning extended FSA from software: An empirical assessment

A number of techniques that infer finite state automata from execution traces have been used to support test and analysis activities. Some of these techniques can produce automata that integrate information about the data-flow, that is, they also represent how data values affect the operations executed by programs. The integration of information about operation sequences and data values into a unique model is indeed conceptually useful to accurately represent the behavior of a program. However, it is still unclear whether handling heterogeneous types of information, such as operation sequences and data values, necessarily produces higher quality models or not. In this paper, we present an empirical comparative study between techniques that infer simple automata and techniques that infer automata extended with information about data-flow. We investigate the effectiveness of these techniques when applied to traces with different levels of sparseness, produced by different software systems. To the best of our knowledge this is the first work that quantifies both the effect of adding data-flow information within automata and the effectiveness of the techniques when varying sparseness of traces.