Andre B. Bondi

Performance Assurance via Software Rejuvenation: Monitoring, Statistics and Algorithms

We present three algorithms for detecting the need for software rejuvenation by monitoring the changing values of a customer-affecting performance metric, such as response time. Applying these algorithms can improve the values of this customer-affecting metric by triggering rejuvenation before performance degradation becomes severe. The algorithms differ in the way they gather and use sample values to arrive at a rejuvenation decision. Their effectiveness is evaluated for different sets of control parameters, including sample size, using simulation. The results show that applying the algorithms with suitable choices of control parameters can significantly improve system performance as measured by the response time

Ensuring stable performance for systems that degrade

A new approach that is useful in identifying and eliminating performance degradation occurring in aging software is proposed. A customer-affecting metric is used to initiate the restoration of such a system to full capacity. A case study is described in which, by simulating an industrial software system, we are able to show that by monitoring a customer-affecting metric and frequently comparing its degradation to the performance objective, we can ensure system stability at a very low cost.

Ensuring system performance for cluster and single server systems

A new approach that is useful in identifying and eliminating performance degradation occurring in aging software is proposed. A customer-affecting metric is used to initiate the restoration of such a system to full capacity. A case study is described in which, by simulating an industrial software system, we are able to show that by monitoring a customer-affecting metric and frequently comparing its degradation to the performance objective, we can ensure system stability at a very low cost.

Experience with Training a Remotely Located Performance Test Team in a Quasi-agile Global Environment

We describe our experience of training a remotely located team of developers and testers to prepare and execute performance tests. The team is located in India. The lead performance engineer and the test project manager are based in New Jersey. The team members had little or no prior experience of performance testing. We describe how we overcame cultural differences and a large time difference to develop a performance testing team that is now functioning well with far less supervision than was required at its inception. Cultural differences included contrasting views on adherence to strict laboratory procedures and assumptions about the prior knowledge, experience, and expectations of working habits of the India-based and New Jersey-based teams. We show how these differences and organizational challenges were overcome with intensive on-site training, the use of twice-daily scrum meetings, the careful designation of team leaders and role players at the remote testing site, and, eventually, the development intensive use of automated tools to execute performance tests and track the results.

Model-Based scalability estimation in inception-phase software architecture

Scalability is one of the crucial nonfunctional requirements that must be evaluated in the Inception Phase of the Rational Unified Process [9]. This is the phase in which the least information is generally available to form a principled evaluation. We demonstrate how an estimate of user scalability can be formed using sequence diagrams of the common user scenarios, together with experimentation (ranging from simple timing measurements to more complex architectural prototypes), published study data, and performance data from baseline systems. Despite being quite inexpensive, the techniques used by our team enabled us to identify and guide corrective actions for major bottlenecks before they became serious design flaws in the Elaboration and Construction phases of the Unified Process. The same techniques also allowed us to quickly evaluate the effects of high-level architecture and technology alternatives on user scalability and response time.

Best practices for writing and managing performance requirements: a tutorial

Performance requirements are one of the main drivers of architectural decisions. Because many performance problems have their roots in architectural decisions, and since poor performance is a principal cause of software project risk, it is essential that performance requirements be developed early in the software lifecycle, and that they be clearly formulated. In this tutorial, we shall look at criteria for high-quality performance requirements, including algebraic consistency, measurability, testability, and linkage to business and engineering needs. While focus of this tutorial is on practice, we shall show how the drafting of performance requirements can be aided by performance modeling. We shall show methods for presenting and managing performance requirements that will improve their chances of being accepted by architects, developers, testers, contract negotiators, and purchasers; and of their being successfully implemented and tested.

Bandwidth usage and network latency in a conveyor system with Ethernet-based communication between controllers

Switched Ethernets are increasingly being used in industrial applications such as the control of conveyor systems and the synchronization of axles in complex machinery. The timely delivery of control messages is a central concern in these applications. We show how a queueing model was used to predict traffic flow and delay in the control networks of a conveyor system. The model indicates that the anticipated level of bandwidth usage will be low on even the most heavily loaded links, and that the resulting end-to-end message latencies will be quite small.

Experience with Performance Engineering Training in Academic and Industrial Environments

We describe our experience of teaching software performance engineering and analysis in academic and industrial settings. There is a need to teach course participants core principles about basic performance modeling and data handling in both settings. These principles are independent of the technologies to which they must be applied and of the measurement and load generation tools that support them. Our experience suggests that it is more difficult for course participants in a university to relate performance engineering to such aspects of the software life cycle as architecture and requirements engineering than it is for industrial participants, because university students are likely lack the related development experience. This is especially true in countries where it is uncommon for mature students to return to graduate school after working outside a university. Our experience also suggests that students who are trained in the natural sciences or who have been rigorously trained in statistics, operations research, or industrial engineering are more likely to devise sound performance tests and be more comfortable with manipulating, interpreting, and analyzing performance data than computer science majors who do not have this type of training. On the other hand, industrial course participants may have had experience of requirements engineering and functional testing processes, but not be used to performing quantitative analyses or devising performance tests to be informative about system behavior and scalability. Some of them may also have considerable experience of using measurement tools and load generators, though not always to the best advantage. Our goal in performance training is to show participants the relationship between performance requirements engineering, testing, architecture, and modeling in principle and in practice. When all participants are involved in the same project in a company, this could be achieved by linking training to a projects performance needs. Otherwise, this can be achieved with exercises that include measuring the resource usage of various applications on their own laptops and explaining their observations. In either case, mentoring of practitioners should include stimulating their curiosity to understand new technologies and to applying various optimization and analysis methods to the issues they encounter.

Challenges with Applying Performance Testing Methods for Systems Deployed on Shared Environments with Indeterminate Competing Workloads: Position Paper

There is a tendency to move production environments from corporate-owned data centers to cloud-based services. Users who do not maintain a private production environment might not wish to maintain a private performance test environment either. The application of performance engineering methods to the development and delivery of software systems is complicated when the form and or parameters of the target deployment environment cannot be controlled or determined. The difficulty of diagnosing the causes of performance issues during testing or production may be increased by the presence of highly variable workloads on the target platform that compete with the application of interest for resources in ways that might be hard to determine. In particular, performance tests might be conducted in virtualized environments that introduce factors influencing customer-affecting metrics (such as transaction response time) and observed resource usage. Observed resource usage metrics in virtualized environments can have different meanings from those in a native environment. Virtual machines may suffer delays in execution. We explore factors that exacerbate these complications. We argue that these complexities reinforce the case for rigorously using software performance engineering methods rather than diminishing it. We also explore possible performance testing methods for mitigating the risk associated with these complexities.

Incorporating Software Performance Engineering Methods and Practices into the Software Development Life Cycle

In many software development projects, attention is only paid to performance concerns after functional testing, when it usually too late to remedy disabling performance problems. Early attention to performance concerns and early planning of performance requirements and performance testing can prevent debacles like the early rollout of healthcare.gov while addressing cross-cutting concerns such as scalability, reliability and, security. Performance engineering methods may be integrated into all phases of the software lifecycle, from the conception of a system to requirements specification, architecture, testing, and finally to production. Performance expectations can be managed by carefully specifying performance requirements. Reviewing the architecture of a system before design and implementation take place reduces the risk of designing a system that contains inherent performance vice. Performance modeling can be used to justify architectural and design decisions and to plan performance tests. The outputs of such performance tests enable us able to identify concurrent programming and other issues that would not be apparent in unit testing. Finally, risk is mitigated by avoiding design antipatterns that undermine scalability and performance.