José Carlos Maldonado

Mutation analysis testing for finite state machines

Proposes the application of the mutation analysis criterion in the context of specifications based on finite state machines. The main concepts of finite state machines and mutation analysis are briefly introduced. An experiment is reported which manually applies mutation analysis to a finite state machine modeling a Class 0 ISO transport protocol specification, using two test-sequence generator criteria-the W method and the TT (transition tours) method. The results obtained are presented, and evidences are given that the use of mutation analysis is effective in this context. Finally, the lines of evolution of the work presented in this paper are briefly discussed.<<ETX>>

Replicating software engineering experiments: addressing the tacit knowledge problem

Recently the awareness of the importance of replicating studies has been growing in the empirical software engineering community. The results of any one study cannot simply be extrapolated to all environments because there are many uncontrollable sources of variation between different environments. In our work, we have reasoned that the availability of laboratory packages for experiments can encourage better replications and complementary studies. However, even with effectively specified laboratory packages, transfer of experimental know-how can still be difficult. A cooperation between Brazilian and American researchers addressing effective running of replications was formed in 1999. One of the specific issues being addressed is the problem of transferring tacit knowledge. We discuss what we learned about the tacit knowledge transfer problem and the evolution of laboratory packages in the description of a replication performed in Brazil using a PBR (Perspective Based Reading) laboratory package; also how further issues are addressed.

Mutation testing applied to validate specifications based on statecharts

The establishment of a low-cost, effective testing and validation strategy has been pursued by many researchers at the program level as well as at the specification level. The application of mutation testing for validating specifications based on statecharts is proposed. A mutation operator set for statecharts, one of the crucial points for effectively applying mutation testing is defined; in this scope these operators can be taken as a fault model. We also provide strategies to abstract the statechart components according to different statechart features that may comprise the testing and validation activity aims, providing in this way mechanisms for the establishment of an incremental, hierarchical, mutation-based testing strategy. Implementation and functional aspects of PROTEUM/ST, a tool under development are also presented.

Knowledge-Sharing Issues in Experimental Software Engineering

Recently the awareness of the importance of replicating studies has been growing in the empirical software engineering community. The results of any one study cannot simply be extrapolated to all environments because there are many uncontrollable sources of variation between different environments.In our work, we have reasoned that the availability of laboratory packages for experiments can encourage better replications and complementary studies. However, even with effectively specified laboratory packages, transfer of experimental know-how can still be difficult. In this paper, we discuss the collaboration structures we have been using in the Readers’ Project, a bilateral project supported by the Brazilian and American national science agencies that is investigating replications and transfer of experimental know-how issues. In particular, we discuss how these structures map to the Nonaka–Takeuchi knowledge sharing model, a well-known paradigm used in the knowledge management literature. We describe an instantiation of the Nonaka–Takeuchi Model for software engineering experimentation, establishing a framework for discussing knowledge sharing issues related to experimental software engineering. We use two replications to illustrate some of the knowledge sharing issues we have faced and discuss the mechanisms we are using to tackle those issues in Readers’ Project.

Mutation Testing for Aspect-Oriented Programs

Mutation testing has been shown to be one of the strongest testing criteria for the evaluation of both programs and test suites. Comprehensive sets of mutants require strong test sets to achieve acceptable testing coverage. Moreover, mutation operators are valuable for the evaluation of other testing approaches. Although its importance has been highlighted for aspect-oriented (AO) programs, there is still a need for a suitable set of mutation operators for AO languages. The quality of the mutation testing itself relies on the quality of such operators. This paper presents the design of a set of mutation operators for AspectJ-based programs. These operators model instances of fault types identified in an extensive survey. The fault types and respective operators are grouped according to the related language features. We also discuss the generalisation of the fault types to AO approaches other than AspectJ and the coverage that may be achieved with the application of the proposed operators. In addition, a cost analysis based on two case studies involving real-world applications has provided us feedback on the most expensive operators, which will support the definition of further testing strategies.

Integration testing using interface mutation

A criterion for assessing the adequacy of test sets during integration testing is proposed. The criterion is based on a testing technique named Interface Mutation. The technique itself is designed to be scalable with the size of the software under test; the size being measured in the number of subsystems integrated. Using Interface Mutation it is possible to assess the adequacy of tests incrementally while integrating various subsystems. Also reported are results from a pilot experiment conducted to study the cost and error defection effectiveness of Interface Mutation.

Mutation testing applied to Estelle specifications

Many researchers have pursued the establishment of a low-cost, effective testing and validation strategy at the program level as well as at the specification level. Mutation Testing is an error-based approach, originally introduced for program testing, that provides testers a systematic way to evaluate how good a given test set is. Some studies have also investigated its use to generate test sets. In this article, the application of Mutation Testing for validating Estelle specifications is proposed. A mutant operator set for Estelle—one of the crucial points for effectively applying Mutation Testing—is defined, addressing: the validation of the behavior of the modules, the communication among modules and the architecture of the specification. In this scope, these operators can be taken as a fault model. Considering this context, a strategy for validating Estelle-based specification is proposed and exemplified using the Alternating-bit protocol.

Proteum/IM 2.0: An Integrated Mutation Testing Environment

Mutation testing has been used mostly at the unit level. To support its application few tools have been developed and used, mainly in the academic environment. Interface Mutation has been proposed aiming at applying mutation at the integration level. A tool named Proteum/IM was implemented to support such criterion. With the definition of the Interface Mutation criterion the tester has the possibility of applying mutation testing concepts throughout the software development. It seems mandatory to have a single, integrated environment that would support mutation-based unit and integration testing. Such environment, which provides facilities to investigate low-cost and incremental testing strategies, is the focus of this paper.

An exploratory study of fault-proneness in evolving aspect-oriented programs

This paper presents the results of an exploratory study on the fault-proneness of aspect-oriented programs. We analysed the faults collected from three evolving aspect-oriented systems, all from different application domains. The analysis develops from two different angles. Firstly, we measured the impact of the obliviousness property on the fault-proneness of the evaluated systems. The results show that 40% of reported faults were due to the lack of awareness among base code and aspects. The second analysis regarded the fault-proneness of the main aspect-oriented programming (AOP) mechanisms, namely pointcuts, advices and intertype declarations. The results indicate that these mechanisms present similar fault-proneness when we consider both the overall system and concern-specific implementations. Our findings are reinforced by means of statistical tests. In general, this result contradicts the common intuition stating that the use of pointcut languages is the main source of faults in AOP.

Control and data flow structural testing criteria for aspect-oriented programs

Although it is claimed that, among other features, aspect-oriented programming (AOP) increases understandability and eases the maintenance burden, this technology cannot provide correctness by itself, and thus it also requires the use of systematic verification, validation and testing (VV&T) approaches. With the purpose of producing high quality software, many approaches to apply structural testing criteria for the unit testing of procedural and object-oriented (OO) programs have been proposed. Nevertheless, until now, few works have addressed the application of such criteria to test aspect-oriented programs. In this paper we define a family of control flow and data flow based testing criteria for aspect-oriented programs inspired by the implementation strategy adopted by AspectJ - an aspect-oriented extension of the Java language - and extending a previous work proposed for Java programs. We propose the derivation of a control and data flow model for aspect-oriented programs based upon the static analysis of the object code (the Java bytecode) resulted from the compilation/weaving process. Using this model, called aspect-oriented def-use graph (AODU), traditional and also aspect-oriented testing criteria are defined (called Control and Data Flow Structural Testing Criteria for Aspect-Oriented Programs - CDSTC-AOP). The main idea is that composition of aspect-oriented programs leads to new crosscutting interfaces in several modules of the system, which must be considered for coverage during structural testing. The implementation of a prototype tool - the JaBUTi/AJ tool - to support the proposed criteria and model is presented along with an example. Also, theoretical and practical questions regarding the CDSTC-AOP criteria are discussed.