Flávio Moreira de Oliveira

Representing software usage models with stochastic automata networks

The benefits of usage models in the statistical testing of software have been recognized by the software engineering community and discrete state formalisms have been largely used to describe such usage models of software. The Markov Chains formalism (MC) is a natural choice to do so. However, in this paper we suggest the use of a more sophisticated discrete state formalism, the Stochastic Automata Networks (SAN). In many problems of the performance evaluation area, the SAN models present advantages over MC models. Therefore, it seems natural to us to verify similar advantages in the modeling of usage models.This paper presents a case study of building a software usage model using SAN formalism. A software tool called DOCSEDITOR is modeled using MC and SAN. The models are compared in terms of number of states, scalability, and readability. It is not the objective of this paper to present a full framework to develop and analyse usage models with SAN, but just to show some evident advantages of the use of SAN instead of MC. In order to do this, the conclusion points out the indexes than can be computed from both models and suggests the next steps on future work.

Test case generation using stochastic automata networks: quantitative analysis

The software engineering community has been using Markov Chains (MC) to describe usage models. We have been working on the use of a more sophisticated discrete state formalism: Stochastic Automata Networks (SAN). SAN is a formalism with the same power of description as MC, however a system in SAN is described as a collection of subsystems described by local states, transitions and synchronizing events, allowing higher modularity and maintainability. We present a description of SAN formalism, as well as quantitative analysis of the modeling examples considering the generation time, quality of the test suites.

Evaluating Load Generation in Virtualized Environments for Software Performance Testing

Before placing a software system into production, it is necessary to guarantee it provides users with a certain level of Quality-of-Service. Intensive performance testing is then necessary to achieve such a level and the tests require an isolated computing environment. Virtualization can therefore play an important role for saving energy costs by reducing the number of servers required to run performance tests and for allowing performance isolation when executing multiple tests in the same computing infrastructure. Load generation is an important component in performance testing as it simulates users interacting with the target application. This paper presents our experience in using a virtualized environment for load generation aimed at performance testing. We measured several performance metrics and varied system load, number of virtual machines per physical resource, and the CPU pinning schema for comparison of virtual and physical machines. The two main findings from our experiments are that physical machines produced steadier and faster response times under heavy load and that the pinning schema is an important aspect when setting up a virtualized environment for load generation.

Evaluating capture and replay and model-based performance testing tools: an empirical comparison

[Context] A variety of testing tools have been developed to support and automate software performance testing activities. These tools may use different techniques, such as Model-Based Testing (MBT) or Capture and Replay (CR). [Goal] For software companies, it is important to evaluate such tools w.r.t. the effort required for creating test artifacts using them; despite its importance, there are few empirical studies comparing performance testing tools, specially tools developed with different approaches. [Method] We are conducting experimental studies to provide evidence about the required effort to use CR-based tools and MBT tools. In this paper, we present our first results, evaluating the effort (time spent) when using LoadRunner and Visual Studio CR-based tools, and the PLeTsPerf MBT tool to create performance test scripts and scenarios to test Web applications, in the context of a collaboration project between Software Engineering Research Center at PUCRS and a technological laboratory of a global IT company. [Results] Our results indicate that, for simple testing tasks, the effort of using a CR-based tool was lower than using an MBT tool, but as the testing complexity increases tasks, the advantage of using MBT grows significantly. [Conclusions] To conclude, we discuss the lessons we learned from the design, operation, and analysis of our empirical experiment.

An experiment using software agents for dialogue analysis in collaborative distance learning

Trends in distance education show a growing emphasis in collaborative learning, stimulating students to exchange ideas and information. A collaborative environment, however, will demand a higher effort from the teacher, who will supervise all discussions among learners, so that they do not deviate from the intended topic for a lesson. Moreover, the information proceeding from interactions among students will provide to the teacher features that allow an individual evaluation of each student and his course. In this way, this paper describes a first experience using a multi-agent architecture that is able to monitor communication tools in a distance-learning group.

An empirical comparison of model-based and capture and replay approaches for performance testing

A variety of testing tools has been developed to support and automate the software testing activity. Some of them may use different techniques such as Model-based Testing (MBT) or Capture and Replay (CR). Model-based Testing is a technique for automatic generation of testing artifacts based on software models. One of the main benefits of using MBT is related to the easiness of maintaining models over code; hence, it is likely that using models as a source for automatic generation of scripts requires less effort and reduces the number of faults. Otherwise, CR-based tools basically record the user interaction with the System Under Test (SUT) and then playback the recorded test. This paper presents our effort on setting up and running an experimental study performed in order to evaluate the effort to use MBT and CR-based tools to generate performance scripts. Thus, we apply an MBT and a CR approaches for the purpose of evaluation with respect to the effort to generate scripts and scenarios from the perspective of the performance testers and the performance test engineers in the context of undergraduates, M.Sc. and Ph.D. students, performance testers and performance test engineers for the generation of performance test scripts and scenarios. Our results indicate that, for simple testing tasks, the effort of using a CR-based tool was lower than using an MBT tool, but as the complexity or size of the activities of the testing tasks increases, the advantage of using MBT increased significantly.

Model comprehension for security risk assessment: an empirical comparison of tabular vs. graphical representations

Tabular and graphical representations are used to communicate security risk assessments for IT systems. However, there is no consensus on which type of representation better supports the comprehension of risks (such as the relationships between threats, vulnerabilities and security controls). Cognitive fit theory predicts that spatial relationships should be better captured by graphs. In this paper we report the results of two studies performed in two countries with 69 and 83 participants respectively, in which we assessed the effectiveness of tabular and graphical representations with respect to extraction correct information about security risks. The experimental results show that tabular risk models are more effective than the graphical ones with respect to simple comprehension tasks and in some cases are more effective for complex comprehension tasks. We explain our findings by proposing a simple extension of Vessey’s cognitive fit theory as some linear spatial relationships could be also captured by tabular models.

PLeTsPerf - A Model-Based Performance Testing Tool

Performance testing is a highly specialized task, since it requires that a performance engineer knows the application to be tested, its usage profile, and the infrastructure where it will execute. Moreover, it requires that testing teams expend a considerable effort and time on its automation. In this paper, we present the PLeTsPerf, a model-based performance testing tool to support the automatic generation of scenarios and scripts from application models. PLetsPerf is a mature tool, developed in collaboration with an IT company, which has been used in several works, experimental studies and pilot studies. We present an example of use to demonstrate the process of generating test scripts and scenarios from UML models to test a Web application. We also present the lessons learned and discuss our conclusions about the use of the tool.

AGES: An Interdisciplinary Space Based on Projects for Software Engineering Learning

The Software Engineering education faces the challenge of qualifying professionals with competence to work in an interdisciplinary way and in teams, with flexibility to perform different roles and capable of adapting to change. This article describes AGES (Agência Experimental de Engenharia de Software), which is part of the curriculum of the new Software Engineering undergraduate program of PUCRS. AGES is a space that allows students to apply their knowledge in an integrated and diversified way. They participate in teams engaged in real projects. We present the assumptions, the implementation process of AGES, its status and lessons learned until now.

An Industrial Experience on using Models to Test Web Service-Oriented Applications

Service-oriented architectures and Web services have been widely adopted by enterprises to pervade integration among software systems. As reliable services are essential to assure that these systems work correctly, formal and systematic testing should be performed. This paper reports the application of a model-based approach to test Web services in the context of real-world applications of a multinational computer technology corporation. The employed approach is called ESG4WSC, in which an event-driven model is provided to support modeling and test case generation, as well as an environment to support the concretization and test execution.