Rafael A. P. Oliveira

Conceptualization and Evaluation of Component-Based Testing Unified with Visual GUI Testing: An Empirical Study

In this paper we present the results of a two-phase empirical study where we evaluate and compare the applicability of automated component-based Graphical User Interface (GUI) testing and Visual GUI Testing (VGT) in the tools GUITAR and a prototype tool we refer to as VGT GUITAR. First, GUI mutation operators are defined to create 18 faulty versions of an application on which both tools are then applied in an experiment. Results from 456 test case executions in each tool show, with statistical significance, that the component-based approach reports more false negatives than VGT for acceptance tests but that the VGT approach reports more false positives for system tests. Second, a case study is performed with larger open source applications, ranging from 8,803-55,006 lines of code. Results show that GUITAR is applicable in practice but has some challenges related to GUI component states. The results also show that VGT GUITAR is currently not applicable in practice and therefore requires further research and development. Based on the studys results we present areas of future work for both test approaches and conclude that the approaches have different benefits and drawbacks. The component-based approach is robust and executes tests faster than the VGT approach, with a factor of 3. However, the VGT approach can perform visual assertions and is perceived more flexible than the component- based approach. These conclusions let us hypothesize that a combination of the two approaches is the most suitable in practice and therefore warrants future research.

Automated Test Oracles: State of the Art, Taxonomies, and Trends

Abstract Test oracle methods have changed significantly over time, which has resulted in clear shifts in the research literature. Over the years, the testing techniques, strategies, and criteria utilized by researchers went through technical developments due to the improvement of technologies and programming languages. Software testing designers, known as testers, currently have several resources to increase their confidence in the software under test correctness. All of these software testing resources are supposed to include a mechanism to decide whether a particular execution is considered a failure or not. In software testing environments, this decision is the responsibility of the test oracle. Despite the evolution and adaptation of testing techniques over more than 30 years, test oracles remain a particular and relevant issue. In this chapter, using literary evidence from a pool of about 300 studies directly related to test oracles, we present a classification of test oracles based on a taxonomy that considers their source of information and notations. Based on this classification, we perform a quantitative analysis to highlight the shifts in (evolution of) research on test oracles. Exploring geographical and quantitative information, we analyzed the maturity of this field using coauthorship networks among studies published between 1978 and 2013. Further, we determine the most prolific authors and their countries, main conferences and journals, supporting tools, and academic efforts and use a comparative analysis between academia and industry. Finally, from these analyses, we draw an analytic reflection about contemporary test oracle approaches and a criticism about oracle trends.

Definition and evaluation of mutation operators for GUI-level mutation analysis

Automated testing has become essential in software industry to meet market demands for faster delivery and higher quality software. Testing is performed on many levels of system abstraction, from tests on source code to Graphical User Interface (GUI) tests. New testing techniques and frameworks are also continuously released to the market. Mutation analysis has been proposed as a way of assessing the quality of these new test techniques/frameworks as well as existing test suites in practice. The analysis is performed by seeding defects, referred to as mutants, into the system under test with the assumption that a technique/test suite of high quality will “kill” the mutants. However, whilst support for mutation analysis exists for test techniques that operate on on lower levels of system abstraction, i.e. method-level mutation operators, the support for GUI-level mutation analysis is currently lacking. In this paper we perform an empirical analysis of 18 GUI-level mutation operators defined in our previous work and compare their efficiency and comprehensiveness to state-of-practice lower level mutation operators. The main findings of our analysis are (1) that traditional method-level mutation operators are not precise enough for GUI-level mutation; (2) the defined GUI-based mutation operators provide comprehensive support for GUI-level mutation; and (3) GUI-based mutation operators can be automated but are challenged by the dependencies between GUI widgets.

Evaluation and assessment of effects on exploring mutation testing in programming courses

Mutation analysis is a testing strategy that consists of using supporting tools to seed artificial faults in the original code of a software under test, generating faulty programs (“mutants”) that are supposed to produce incorrect outputs. Novice programmers suffer of a wide range of deficits due to defective training processes. We argue that the incorporation of experiences on mutation testing in programming courses adds valuable knowledge to the learning process. In this paper we evaluate the effects of using mutation testing to improve the learning process of students in programming courses. We present results of experiments and analysis involving undergraduate students. These experiments are the continuation of a previous work in which we raise empirical evidences that the adequate incorporation of mutation testing in programming courses contributes to form an effective environment that fosters learning. To do so, we provide a mutation testing tool to promote the practice of mutation testing by novice programmers. Through practical experiences and several analysis survey we measured the effects of using the mutation testing criterion to teach programming. In addition, we collected the opinion of senior students who already knew mutation testing concepts about their opinion on the usage of mutation concepts to teach novice programmers. Our findings reveal that the effective use of mutation analysis concepts contributes to the learning process, making students see the code as a product under development that is the result of a careful manual coding process which they need for measuring and predicting the effect of each command. The main contributions discussed in this paper are: (1) presenting results of an empirical analysis involving undergraduate students, thus giving us preliminary evidence on the effects of the novel practice; (2) exposing possible practices to explore mutation testing in programming classes, highlighting the limitations and strengths of such strategy; and (3) a mutation testing tool for educational purposes.

What a Long, Strange Trip It's Been: Past, Present, and Future Perspectives on Software Testing Research

Over the past 25 years the Brazilian Symposium on Software Engineering (SBES) has evolved to become the most important event on software engineering in Brazil. Throughout these years, SBES has gathered a large body of studies in software testing. Aimed at providing an insightful understanding of what has already been published in such event, we synthesized its rich 25-year history of research on software testing. Using information drawn from this overview we attempted to highlight which types of study have been the most applied for conveying software testing efforts. We also devised a co-authorship network to obtain a birds-eye view of which research groups and scholars have been the most prolific ones. Moreover, by performing a citation analysis of the selected studies we set out to ascertain the importance of SBES in a wider scenario. Finally, borne out by the information extracted from the studies, we shed some light on the state-of-the-art of software testing in Brazil and provide an outlook on its foreseeable future.

Content-Based Image Retrieval of 3D Cardiac Models to Aid the Diagnosis of Congestive Heart Failure by Using Spectral Clustering

This paper describes a novel application of Content-Based Image Retrieval (CBIR) to search a medical database consisting of 3D models for diagnosis purposes. The 3D models, which are generated using Magnetic Resonance Imaging and include depth information, are used to search for similarity a database of 3D annotated medical cases using their pairwise feature similarity. The 3D models consist of both local and global feature descriptors that consider the surface of the 3D model and the overall geometry of the medical artifact. The models are then matched using spectral clustering that embeds the Euclidean distance for affinity and partitions the models into two groups, Congestive Heart Failure (CHF) and non-CHF. This suffices to demarcate using pairwise similarity the existence of CHF for the left ventricle. Experimental results using thirty 3D models show the utility of the new 3D method compared to existing methods. In particular, the novel method yields 83% overall accuracy.

A Framework Based on Learning Techniques for Decision-making in Self-adaptive Software

The development of Self-adaptive Software (SaS) presents specific innovative features compared to traditional ones since this type of software constantly deals with structural and/or behavioral changes at runtime. Capabilities of human administration are showing a decrease in relative effectiveness, since some tasks have been difficult to manage introducing potential problems, such as change management and simple human error. Self-healing systems, a system class of SaS, have emerged as a feasible solution in contrast to management complexity, since such system often combines machine learning techniques with control loops to reduce the number of situations requiring human intervention. This paper presents a framework based on learning techniques and the control loop (MAPE-K) to support the decision-making activity for SaS. In addition, it is noteworthy that this framework is part of a wider project developed by the authors of this paper in previous work (i.e., reference architecture for SaS [1]). Aiming to present the viability of our framework, we have conducted a case study using a flight plan module for Unmanned Aerial Vehicles. The results have shown an environment accuracy of about 80%, enabling us to project good perspectives of contribution to the SaS area and other domains of software systems, and enabling knowledge sharing and technology transfer from academia to industry. Keywords-Self-adaptive software; Reference Architecture; Framework; Learning Techniques; Decision-making.

Automating Test Oracles for Systems with Complex Outputs

In Software Testing, systems with complex outputs (GUIs, web applications, and Text-to-speech (TTS) systems) make the test automation a difficult job and may require from the tester a large amount of time to determine whether the current output is correct. Even so, there is still no known general method to define test oracles for such systems. Regarding TTS systems, in which audio files are given as output, the literature describes several techniques useful for measure their quality, but they mostly involve opinion scores, informal interpretations and human intervention. This Ph.D. Research aim to use test oracles to deal with this problem, extending the framework O-FIm that uses CBIR concepts to automate test oracles. Using feature extractors implemented specifically for systems with complex outputs (image and audio), this research provides an effective automated test oracle technique, supporting the quality assessment of such systems. We expect this new approach to reduce the human efforts of testing systems with complex outputs.