Breno Miranda

Social coverage for customized test adequacy and selection criteria

Test coverage information can be very useful for guiding testers in enhancing their test suites to exercise possible uncovered entities and in deciding when to stop testing. However, for complex applications that are reused in different contexts and for emerging paradigms (e.g., component-based development, service-oriented architecture, and cloud computing), traditional coverage metrics may no longer provide meaningful information to help testers on these tasks. Various proposals are advocating to leverage information that come from the testing community in a collaborative testing approach. In this work we introduce a coverage metric, the Social Coverage, that customizes coverage information in a given context based on coverage data collected from similar users. To evaluate the potential of our proposed approach, we instantiated the social coverage metric in the context of a real world service oriented application. In this exploratory study, we were able to predict the entities that would be of interest for a given user with an average precision of 97% and average recall of 75%. Our results suggest that, in similar environments, social coverage can provide a better support to testers than traditional coverage.

A proposal for revisiting coverage testing metrics

Test coverage information can be very useful for guiding testers in enhancing their test suites to exercise possible uncovered entities and in deciding when to stop testing. Since the concept of test criterion was born, several contributions have been made by both academia and industry in the definition and adaptation of adequacy criteria aiming at ensuring the discovery of more failures. Numerous contributions have also been done in the development of coverage tools. However, for complex applications that are reused in different contexts and for emerging paradigms (e.g., component-based development, service-oriented architecture, and cloud computing), traditional coverage metrics may no longer provide meaningful information to help testers on these tasks. Inspired by the idea of relative coverage this research focuses on the introduction of meaningful coverage metrics to cope with the challenges imposed by the current programming paradigms as well as on the definition of a theoretical framework for the development of relative coverage metrics.

Does code coverage provide a good stopping rule for operational profile based testing

We introduce a new coverage measure, called the operational coverage, which is customized to the usage profile (count spectrum) of the entities to be covered. Operational coverage is proposed as an adequacy criterion for operational profile based testing, i.e., to assess the thoroughness of a black box test suite derived from the operational profile. To validate the approach we study the correlation between operational coverage of branches, statements, and functions, and the probability that the next test input will not fail. On the three subjects considered, we observed a moderate correlation in all cases (except a low correlation for function coverage for one subject), and consistently better results than traditional coverage measure.

FAST approaches to scalable similarity-based test case prioritization

Many test case prioritization criteria have been proposed for speeding up fault detection. Among them, similarity-based approaches give priority to the test cases that are the most dissimilar from those already selected. However, the proposed criteria do not scale up to handle the many thousands or even some millions test suite sizes of modern industrial systems and simple heuristics are used instead. We introduce the FAST family of test case prioritization techniques that radically changes this landscape by borrowing algorithms commonly exploited in the big data domain to find similar items. FAST techniques provide scalable similarity-based test case prioritization in both white-box and black-box fashion. The results from experimentation on real world C and Java subjects show that the fastest members of the family outperform other black-box approaches in efficiency with no significant impact on effectiveness, and also outperform white-box approaches, including greedy ones, if preparation time is not counted. A simulation study of scalability shows that one FAST technique can prioritize a million test cases in less than 20 minutes.

What paper types are accepted at the international conference on software engineering

With the aim of identifying good structures and examples for papers in the software engineering field, we conducted a study of the type of papers accepted along four decades in the Research Track of the International Conference on Software Engineering (ICSE). We used for this purpose a categorization scheme for Software Engineering papers that was obtained by merging, extending and revising a few existing paper scheme proposals. This paper summarizes some outcomes relative to what topics and problems are addressed, what types of contribution are presented and how they are validated. Insights from the study could help ICSE authors, reviewers and conference organizers in focusing and improving future efforts.

Adaptive coverage and operational profile-based testing for reliability improvement

We introduce covrel, an adaptive software testing approach based on the combined use of operational profile and coverage spectrum, with the ultimate goal of improving the delivered reliability of the program under test. Operational profile-based testing is a black-box technique that selects test cases having the largest impact on failure probability in operation, as such, it is considered well suited when reliability is a major concern. Program spectrum is a characterization of a programs behavior in terms of the code entities (e.g., branches, statements, functions) that are covered as the program executes. The driving idea of covrel is to complement operational profile information with white-box coverage measures based on count spectra, so as to dynamically select the most effective test cases for reliability improvement. In particular, we bias operational profile-based test selection towards those entities covered less frequently. We assess the approach by experiments with 18 versions from 4 subjects commonly used in software testing research, comparing results with traditional operational and coverage testing. Results show that exploiting operational and coverage data in a combined adaptive way actually pays in terms of reliability improvement, with covrel overcoming conventional operational testing in more than 80% of the cases.

Towards Automated Deployment of Self-adaptive Applications on Hybrid Clouds (Short Paper)

Cloud computing promises high dynamism, flexibility, and elasticity of applications at lower infrastructure costs. However, resource management, portability, and interoperability remain a challenge for cloud application users, since the current major cloud application providers have not converged to a standard interface, and the deployment supporting tools are highly heterogeneous. Besides, by their very nature, cloud applications bring serious traceability, security and privacy issues. This position paper describes a research thread on an extensible Domain Specific Language (DSL), a platform for the automated deployment , and a generic architecture of an ops application manager for self-adaptive distributed applications on hybrid cloud infrastructures. The idea is to overcome the cited limitations by empowering the cloud applications with self-configuration, self-healing, and self-protection capabilities. Such autonomous governance can be achieved by letting cloud users define their policies concerning security, data protection, dependability and functional compliance behavior using the proposed DSL. Real world trials in different application domains are discussed

An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing

While the relation between code coverage measures and fault detection is actively studied, only few works have investigated the correlation between measures of coverage and of reliability. In this work, we introduce a novel approach to measuring code coverage, called the operational coverage, that takes into account how much the program’s entities are exercised so to reflect the profile of usage into the measure of coverage. Operational coverage is proposed as (i) an adequacy criterion, i.e., to assess the thoroughness of a black box test suite derived from the operational profile, and as (ii) a selection criterion, i.e., to select test cases for operational profile-based testing. Our empirical evaluation showed that operational coverage is better correlated than traditional coverage with the probability that the next test case derived according to the user’s profile will not fail. This result suggests that our approach could provide a good stopping rule for operational profile-based testing. With respect to test case selection, our investigations revealed that operational coverage outperformed the traditional one in terms of test suite size and fault detection capability when we look at the average results.