Elder de M. Rodrigues

Performance testing modeling: an empirical evaluation of DSL and UML-based approaches

Performance testing modeling is a relative new research field. Researches investigating how to apply models to document performance testing information essentially started to be reported in the last decade. Motivated by the lack of a standard to represent performance testing information, our research group, in collaboration with an IT company, proposed a UML approach and lately a Domain-Specific Language (DSL) to support performance testing modeling. The goal of this paper is to show how we support our partner company on the decision process to replace UML by a DSL, hence we designed and conducted an experimental study to provide evidence about the benefits and drawbacks when using UML or DSL for modeling performance testing. In this paper, we report an in vitro experiment, where the subjects designed UML models and DSL models, for the purpose of evaluation with respect to the effort and suitability, from the perspective of the performance testers and the performance engineers in the context of industry and academia for modeling performance testing. Our results indicate that, for performance modeling, effort using a DSL was lower than using UML. Our statistical analysis showed that the results were valid, i.e., that to design performance testing models using our DSL is better than using UML.

Canopus: A Domain-Specific Language for Modeling Performance Testing

Despite all the efforts to reduce the cost of the testing phase in software development, it is still one of the most expensive phases. In order to continue to minimize those costs, in this paper, we propose a Domain-Specific Language (DSL), built on top of MetaEdit+ language workbench, to model performance testing for web applications. Our DSL, called Canopus, was developed in the context of a collaboration1 between our university and a Technology Development Laboratory (TDL) from an Information Technology (IT) company. We present, in this paper, the Canopus metamodels, its domain analysis, a process that integrates Canopus to Model-Based Performance Testing, and applied it to an industrial case study.

Evaluating capture and replay and model-based performance testing tools: an empirical comparison

[Context] A variety of testing tools have been developed to support and automate software performance testing activities. These tools may use different techniques, such as Model-Based Testing (MBT) or Capture and Replay (CR). [Goal] For software companies, it is important to evaluate such tools w.r.t. the effort required for creating test artifacts using them; despite its importance, there are few empirical studies comparing performance testing tools, specially tools developed with different approaches. [Method] We are conducting experimental studies to provide evidence about the required effort to use CR-based tools and MBT tools. In this paper, we present our first results, evaluating the effort (time spent) when using LoadRunner and Visual Studio CR-based tools, and the PLeTsPerf MBT tool to create performance test scripts and scenarios to test Web applications, in the context of a collaboration project between Software Engineering Research Center at PUCRS and a technological laboratory of a global IT company. [Results] Our results indicate that, for simple testing tasks, the effort of using a CR-based tool was lower than using an MBT tool, but as the testing complexity increases tasks, the advantage of using MBT grows significantly. [Conclusions] To conclude, we discuss the lessons we learned from the design, operation, and analysis of our empirical experiment.

An empirical comparison of model-based and capture and replay approaches for performance testing

A variety of testing tools has been developed to support and automate the software testing activity. Some of them may use different techniques such as Model-based Testing (MBT) or Capture and Replay (CR). Model-based Testing is a technique for automatic generation of testing artifacts based on software models. One of the main benefits of using MBT is related to the easiness of maintaining models over code; hence, it is likely that using models as a source for automatic generation of scripts requires less effort and reduces the number of faults. Otherwise, CR-based tools basically record the user interaction with the System Under Test (SUT) and then playback the recorded test. This paper presents our effort on setting up and running an experimental study performed in order to evaluate the effort to use MBT and CR-based tools to generate performance scripts. Thus, we apply an MBT and a CR approaches for the purpose of evaluation with respect to the effort to generate scripts and scenarios from the perspective of the performance testers and the performance test engineers in the context of undergraduates, M.Sc. and Ph.D. students, performance testers and performance test engineers for the generation of performance test scripts and scenarios. Our results indicate that, for simple testing tasks, the effort of using a CR-based tool was lower than using an MBT tool, but as the complexity or size of the activities of the testing tasks increases, the advantage of using MBT increased significantly.

PLeTsPerf - A Model-Based Performance Testing Tool

Performance testing is a highly specialized task, since it requires that a performance engineer knows the application to be tested, its usage profile, and the infrastructure where it will execute. Moreover, it requires that testing teams expend a considerable effort and time on its automation. In this paper, we present the PLeTsPerf, a model-based performance testing tool to support the automatic generation of scenarios and scripts from application models. PLetsPerf is a mature tool, developed in collaboration with an IT company, which has been used in several works, experimental studies and pilot studies. We present an example of use to demonstrate the process of generating test scripts and scenarios from UML models to test a Web application. We also present the lessons learned and discuss our conclusions about the use of the tool.

An Industrial Experience on using Models to Test Web Service-Oriented Applications

Service-oriented architectures and Web services have been widely adopted by enterprises to pervade integration among software systems. As reliable services are essential to assure that these systems work correctly, formal and systematic testing should be performed. This paper reports the application of a model-based approach to test Web services in the context of real-world applications of a multinational computer technology corporation. The employed approach is called ESG4WSC, in which an event-driven model is provided to support modeling and test case generation, as well as an environment to support the concretization and test execution.

Architectural Evolution of a Software Product Line: an experience report

This work presents an experience report on the architectural decisions taken in the evolution of a Software Product Line (SPL) of Model-based Testing tools (PLeTs). This SPL was partially designed and developed with the intention of minimizing effort and time-to-market during the development of a family of performance testing tools. With the evolution of our research and the addition of new features to the SPL, we identified limitations in the initial architectural design of PLeTs’ components, which led us to redesign its Software Product Line Architecture (SPLA). In this paper, we discuss the main issues that led to changes in our SPLA, as well as present the design decisions that facilitate its evolution in the context of an industrial environment. We will also report our experiences on architecture modifications in the evolution of our SPL with the intention of allowing easier maintenance in a volatile development environment.

Research Notes on the Architectural Evolution of a Software Product Line

This work presents an experience report on the architectural decisions taken in the evolution of a Software Product Line (SPL) of Model-based Testing tools (PLeTs). This SPL was partially designed and developed with the intention of minimizing effort and time-to-market during the development of a family of performance testing tools. With the evolution of our research and the addition of new features to the SPL, we identified limitations in the initial architectural design of PLeTs’ components, which led us to redesign its Software Product Line Architecture (SPLA). In this paper, we discuss the main issues that led to changes in our SPLA, as well as present the design decisions that facilitate its evolution in the context of an industrial environment. We will also report our experiences on architecture modifications in the evolution of our SPL with the intention of allowing easier maintenance in a volatile development environment.