Manoj K. Nambiar

Recovery from Failures Due to Mandelbugs in IT Systems

Several studies have been carried out on software bugs analysis and classification for life and mission critical systems, which include reproducible bugs called Bohrbugs, and hard to reproduce bugs called Mandelbugs. Although software reliability in IT systems has been studied for years, there are only a few formal analytic models for recovery from Mandelbugs. This paper discusses in detail several real cases of Mandelbugs and presents a simple flowchart which describes the recovery processes implemented in IT systems for a large variety of Mandelbugs. The flowchart is based on more than 10 IT systems that are running in production. The paper then presents a closed-form expression of the mean time to recovery from these bugs. Measures of interest including mean time to recovery and system unavailability are computed. A numerical and parametric sensitivity analysis of the model parameters are carried out. This analysis allows the designer to find out important parameter(s) for the recovery from failures due to Mandelbugs.

Extrapolation of SQL query elapsed response time at application development stage

In a typical database application development, requirement is to optimize SQL queries to meet service level agreements (SLA); the optimized queries are tested on the application development database which is some fraction of the production database. As time progresses the database grows and the earlier optimized queries may not hold SLA anymore. Once the application is launched and deployed, it becomes difficult and expensive to modify the SQL queries. In this paper, we have discussed a model for predicting the SQL query cost and hence the SQL query elapsed response time with the growth of the database at application development time. We have identified and discussed the database statistics which can impact the SQL query cost with increase in the database size and how these can be used to predict ERT. We have tested the model on Oracle 10g and have presented the results in the paper.

Predicting performance in the presence of software and hardware resource bottlenecks

Scalability of a multi-tier enterprise system is limited by the presence of software and hardware resource bottlenecks. These bottlenecks typically occur at larger number of users. It would help enterprise applications significantly if these bottlenecks are known a-priori during the performance testing itself. This paper deals with predicting the performance of such systems and models an application in terms of a two layer queuing network consisting of software resources and hardware resources. The software modules which require exclusive access by a thread are modeled as a queuing resource and other modules are treated as delay resources in the software queuing network. This network in turn uses a hardware queuing network consisting of resources such as CPU, disk and network. The proposed solution is augmented with additional constraints to ensure that the solution converges at a large number of users. Further, the proposed solution is capable of modeling multi-class requests with critical section and pooling of resources e.g., connection pool or thread pool. We validate the proposed solution with actual experimental results using sample programs and observe that the model is able to predict throughput and resource utilization with close to 90% accuracy.

Recovery From Software Failures Caused by Mandelbugs

Software failures are still a major concern in mission- and enterprise-critical contexts, despite significant efforts spent in software testing. In fact, while software testing is effective against easily-reproducible bugs (Bohrbugs), it is considerably less suitable for dealing with bugs that lead to hard-to-reproduce failures (Mandelbugs). On the positive side, the elusive nature of Mandelbugs provides opportunities for failure recovery, which are investigated in this paper. Based on real cases of Mandelbugs in eleven Information Technology (IT) systems running in production, the paper proposes a model that describes the recovery processes in IT systems. It then presents closed-form expressions, and a numerical analysis, of the mean time to recovery, and the software (un)availability. This analysis allows the designer to compare recovery strategies, as well as to determine the parameters having a high influence on the efficacy of recovery from failures caused by Mandelbugs.

Performance Extrapolation for Load Testing Results of Mixture of Applications

Load testing of IT applications faces the challenge of providing high quality test results that would represent the performance in production like scenarios, without incurring high cost of commercial load testing tools. It would help IT projects to be able to test with a small number of users and extrapolate to scenarios with much larger number of users. Such an extrapolation strategy when applied to mixture of application workloads running on a shared server environment must take into consideration application characteristics (CPU/IO intensive, memory bound) as well the server capabilities. The goal is to predict the performance of mixture workload, the maximum throughput offered by the application mix and the maximum number of users supported by the system before the throughput starts degrading. In this paper, we propose an extrapolation strategy that analyses a system workload mix based on its service demand on various resources and extrapolates its performance using simple empirical modeling techniques. Moreover, its ability to extrapolate throughput of an application mixture even if there is a change in the mixture, can help in capacity planning of the system.

A New Algorithm for Measuring Available Bandwidth in a Wide Area Network

Available bandwidth is a very important characteristic of a link across a wide area network. In this paper we dis- cuss a new method to measure available bandwidth of a link across a wide area network. This method does not introduce large traffic into the network during measurement. Also, it does not require both ends of a link to run the algorithm. Ours is a simple to use algorithm which runs at one end of the link and needs only the IP address of the other end.

Network Emulation and Simultaneous Monitoring of Web Based Applications Performance Using ScrutiNem

Performance testing of a multi-tiered Web application in real WAN is not a trivial job. In this circumstance a tool which brings WAN cloud to a local area network makes the life of a developer/tester easier. However, mere emulation of WAN inside a LAN does not help a tester, unless he gets to know the performance of the applications in some numbers. Therefore, he needs to put a monitoring tool in the network too. This means a lot of manual handling of operations such as routing, emulating, port mirroring of switch and measuring the performance of applications simultaneously. We can well understand that this is a huge job for a tester. Can we bring network emulation and application performance monitoring tool together in a single box? This would become a great help for a developer/tester, because in that case he just needs only one machine to configure for the same. This paper investigates this problem and gives a solution thereafter.

Service demand modeling and performance prediction with single-user tests

Abstract Performance load tests of online transaction processing (OLTP) applications are expensive in terms of manpower, time and costs. Alternative performance modeling and prediction tools are required to generate accurate outputs with minimal input sample points. Service Demands (time needed to serve 1 request at queuing stations) are typically needed as inputs by most performance models. However, as service demands vary as a function of workload (load dependent service demands), models that input singular service demands produce erroneous predictions. The alternative, which is to collect service demands at varying workloads, requires time and resource intensive load tests to estimate multiple sample points—which defeats the purpose of performance modeling for industrial use. In this paper, we propose a service demand model as a function of concurrency that can be estimated with a single-user test. Further, we analyze multiple CPU performance metrics (cache hits/misses, branch prediction, context switches and so on) using Principal Component Analysis (PCA) to extract a regression function of service demand with increasing workloads. We use the service demand models as input to performance prediction algorithms such as Mean Value Analysis (MVA), to accurately predict throughput at varying workloads. This service demand prediction model uses CPU hardware counters, which is used in conjunction with a modified version of MVA with single-user service demand inputs. The predicted throughput values are within 9 % deviation with measurements procured for a variety of application/hardware configurations. Such a service demand model is a step towards reducing reliance on conventional load testing for performance assurance.

Model Driven Software Performance Engineering: Current Challenges and Way Ahead

Performance model solvers and simulation engines have been around for more than two decades. Yet, performance modeling has not received wide acceptance in the software industry, unlike pervasion of modeling and simulation tools in other industries. This paper explores underlying causes and looks at challenges that need to be overcome to increase utility of performance modeling, in order to make critical decisions on software based products and services. Multiple real-world case studies and examples are included to highlight our viewpoints on performance engineering. Finally, we conclude with some possible directions the performance modeling community could take, for better predictive capabilities required for industrial use.

Design of a new algorithm for WAN disconnection emulation and implementing it in WANem

Network availability is a very important need of any production environment. Unfortunately due to many reasons a link in a wide area network does not have consistent availability for a long period of time. Hence an application needs to be designed considering this unreliability factor of a wide area network. However, such network instabilities are not seen in a typical LAN based development environment. This is the reason why most of the applications developed in a LAN based development environment perform very badly while in production in a Wide Area Network. It is observed that -- testing robustness of such an application in a wide area network is a difficult task. In this paper a new method which can emulate network availability of a Wide Area Network (WAN) inside the LAN is proposed. Using proposed method, one can, therefore, create a WAN availability test condition within the LAN itself. This way an application can be tested as if it is running in an actual production environment.