Diwakar Krishnamurthy

Characterizing the Scalability of a Large Web-Based Shopping System

This article presents an analysis of five days of workload data from a large Web-based shopping system. The multitier environment of this Web-based shopping system includes Web servers, application servers, database servers, and an assortment of load-balancing and firewall appliances. We characterize user requests and sessions and determine their impact on system performance scalability. The purpose of our study is to assess scalability and support capacity planning exercises for the multitier system. We find that horizontal scalability is not always an adequate mechanism for supporting increased workloads and that personalization and robots can have a significant impact on system scalability.

A Synthetic Workload Generation Technique for Stress Testing Session-Based Systems

Enterprise applications are often business critical but lack effective synthetic workload generation techniques to evaluate performance. These workloads are characterized by sessions of interdependent requests that often cause and exploit dynamically generated responses. Interrequest dependencies must be reflected in synthetic workloads for these systems to exercise application functions correctly. This poses significant challenges for automating the construction of representative synthetic workloads and manipulating workload characteristics for sensitivity analyses. This paper presents a technique to overcome these problems. Given request logs for a system under study, the technique automatically creates a synthetic workload that has specified characteristics and maintains the correct interrequest dependencies. The technique is demonstrated through a case study involving a TPC-W e-commerce system. Results show that incorrect performance results can be obtained by neglecting interrequest dependencies, thereby highlighting the value of our technique. The study also exploits our technique to investigate the impact of several workload characteristics on system performance. Results establish that high variability in the distributions of session length, session idle times, and request service times can cause increased contention among sessions, leading to poor system responsiveness. To the best of our knowledge, these are the first results of this kind for a session-based system. We believe our technique is of value for studies where fine control over workload is essential

A model-based approach for testing the performance of web applications

Poor performance of Web-based systems can adversely impact the profitability of enterprises that rely on them. As a result, effective performance testing techniques are essential for understanding whether a Web-based system will meet its performance objectives when deployed in the real world. The workload of a Web-based system has to be characterized in terms of sessions; a session being a sequence of inter-dependent requests submitted by a single user. Dependencies arise because some requests depend on the responses of earlier requests in a session. To exercise application functions in a representative manner, these dependencies should be reflected in the synthetic workloads used to test Web-based systems. This makes performance testing a challenge for these systems. In this paper, we propose a model-based approach to address this problem. Our approach uses an application model that captures the dependencies for a Web-based system under study. Essentially, the application model can be used to obtain a large set of valid request sequences representing how users typically interact with the application. This set of sequences can be used to automatically construct a synthetic workload with desired characteristics. The application model provides an indirection which allows a common set of workload generation tools to be used for testing different applications. Consequently, less effort is needed for developing and maintaining the workload generation tools and more effort can be dedicated towards the performance testing process.

DEC: Service Demand Estimation with Confidence

We present a new technique for predicting the resource demand requirements of services implemented by multitier systems. Accurate demand estimates are essential to ensure the efficient provisioning of services in an increasingly service-oriented world. The demand estimation technique proposed in this paper has several advantages compared with regression-based demand estimation techniques, which many practitioners employ today. In contrast to regression, it does not suffer from the problem of multicollinearity, it provides more reliable aggregate resource demand and confidence interval predictions, and it offers a measurement-based validation test. The technique can be used to support system sizing and capacity planning exercises, costing and pricing exercises, and to predict the impact of changes to a service upon different service customers.

A Model of Storage I/O Performance Interference in Virtualized Systems

In this paper, we propose simple performance models to predict the impact of consolidation on the storage I/O performance of virtualized applications. We use a measurement-based approach based on tools such as blktrace and tshark for storage workload characterization in a commercial virtualized solution, namely VMware ESX server. Our approach allows a distinct characterization of read/write performance attributes on a per request level and provides valuable information for parameterization of storage I/O performance models. In particular, based on measures of quantities such as the mean queue-length seen upon arrival by requests, we define simple linear prediction models for the throughput, response times, and mix of read/write requests in consolidation based only on information collected in isolation experiments for the individual virtual machines.

Performance models of storage contention in cloud environments

We propose simple models to predict the performance degradation of disk requests due to storage device contention in consolidated virtualized environments. Model parameters can be deduced from measurements obtained inside Virtual Machines (VMs) from a system where a single VM accesses a remote storage server. The parameterized model can then be used to predict the effect of storage contention when multiple VMs are consolidated on the same server. We first propose a trace-driven approach that evaluates a queueing network with fair share scheduling using simulation. The model parameters consider Virtual Machine Monitor level disk access optimizations and rely on a calibration technique. We further present a measurement-based approach that allows a distinct characterization of read/write performance attributes. In particular, we define simple linear prediction models for I/O request mean response times, throughputs and read/write mixes, as well as a simulation model for predicting response time distributions. We found our models to be effective in predicting such quantities across a range of synthetic and emulated application workloads.

Web workload generation challenges - an empirical investigation

Workload generators are widely used for testing the performance of Web‐based systems. Typically, these tools are also used to collect measurements such as throughput and end‐user response times that are often used to characterize the QoS provided by a system to its users. However, our study finds that Web workload generation is more difficult than it seems. In examining the popular RUBiS client generator, we found that reported response times could be grossly inaccurate, and that the generated workloads were less realistic than expected, causing server scalability to be incorrectly estimated. Using experimentation, we demonstrate how the Java virtual machine and the Java network library are the root causes of these issues. Our work serves as an example of how to verify the behavior of a Web workload generator. Copyright

Predictive modelling of SAP ERP applications: challenges and solutions

Analytic performance models are being increasingly used to support system runtime optimization. This paper considers the modelling features needed to predict the response time behaviour of an industrial enterprise resource planning (ERP) application, SAP ERP. A number of studies have reported modelling success with the application of basic product-form Queueing Network Models (QNMs) to multi-tier systems. Such QNMs are often preferred in the context of optimization studies due to the low computational costs of their solution. However, we show that these simple models do not support many important features required to accurately characterize industrial applications such as ERP systems. Specifically, our results indicate that software threading levels, asynchronous database calls, priority scheduling, multiple phases of processing, and the parallelism offered by multi-core processors all have a significant impact on response time that cannot be neglected. Starting from these observations, the paper shows that Layered Queueing Models (LQMs) are a robust alternative to basic QNMs, while still enjoying analytical solution algorithms that facilitate their integration in optimization studies. A case study for a sales and distribution workload demonstrates that many of the features supported by LQMs are critical for achieving good prediction accuracy. Results show that, remarkably, all of the features we considered that are not captured by basic product-form QNMs are needed to predict mean response times to within 15% of measured values for a wide range of load levels. If any key feature is absent, the mean response time estimates could differ by 36% to 117% compared to the measured values, thus making the case that such non-product-form modelling features are needed for complex real-world applications.

WAM—The Weighted Average Method for Predicting the Performance of Systems with Bursts of Customer Sessions

Predictive performance models are important tools that support system sizing, capacity planning, and systems management exercises. We introduce the Weighted Average Method (WAM) to improve the accuracy of analytic predictive performance models for systems with bursts of concurrent customers. WAM considers the customer population distribution at a system to reflect the impact of bursts. The WAM approach is robust with respect to distribution functions, including heavy-tail-like distributions, for workload parameters. We demonstrate the effectiveness of WAM using a case study involving a multitier TPC-W benchmark system. To demonstrate the utility of WAM with multiple performance modeling approaches, we developed both Queuing Network Models and Layered Queuing Models for the system. Results indicate that WAM improves prediction accuracy for bursty workloads for QNMs and LQMs by 10 and 12 percent, respectively, with respect to a Markov Chain approach reported in the literature.

IO performance prediction in consolidated virtualized environments

We propose a trace-driven approach to predict the performance degradation of disk request response times due to storage device contention in consolidated virtualized environments. Our performance model evaluates a queueing network with fair share scheduling using trace-driven simulation. The model parameters can be deduced from measurements obtained inside Virtual Machines (VMs) from a system where a single VM accesses a remote storage server. The parameterized model can then be used to predict the effect of storage contention when multiple VMs are consolidated on the same virtualized server. The model parameter estimation relies on a search technique that tries to estimate the splitting and merging of blocks at the the Virtual Machine Monitor (VMM) level in the case of multiple competing VMs. Simulation experiments based on traces of the Postmark and FFSB disk benchmarks show that our model is able to accurately predict the impact of workload consolidation on VM disk IO response times.