Stephen Dawson

Estimating service resource consumption from response time measurements

We propose a linear regression method and a maximum likelihood technique for estimating the service demands of requests based on measurement of their response times instead of their CPU utilization. Our approach does not require server instrumentation or sampling, thus simplifying the parameterization of performance models. The benefit of this approach is further highlighted when utilization measurement is difficult or unreliable, such as in virtualized systems or for services controlled by third parties. Both experimental results from an industrial ERP system and sensitivity analyses based on simulations indicate that the proposed methods are often much more effective for service demand estimation than popular utilization based linear regression methods. In particular, the maximum likelihood approach is found to be typically two to five times more accurate than utilization based regression, thus suggesting that estimating service demands from response times can help in improving performance model parameterization.

Markovian Workload Characterization for QoS Prediction in the Cloud

Resource allocation in the cloud is usually driven by performance predictions, such as estimates of the future incoming load to the servers or of the quality-of-service(QoS) offered by applications to end users. In this context, characterizing web workload fluctuations in an accurate way is fundamental to understand how to provision cloud resources under time-varying traffic intensities. In this paper, we investigate the Markovian Arrival Processes (MAP) and the related MAP/MAP/1 queueing model as a tool for performance prediction of servers deployed in the cloud. MAPs are a special class of Markov models used as a compact description of the time-varying characteristics of workloads. In addition, MAPs can fit heavy-tail distributions, that are common in HTTP traffic, and can be easily integrated within analytical queueing models to efficiently predict system performance without simulating. By comparison with traced riven simulation, we observe that existing techniques for MAP parameterization from HTTP log files often lead to inaccurate performance predictions. We then define a maximum likelihood method for fitting MAP parameters based on data commonly available in Apache log files, and a new technique to cope with batch arrivals, which are notoriously difficult to model accurately. Numerical experiments demonstrate the accuracy of our approach for performance prediction of web systems.

DEC: Service Demand Estimation with Confidence

We present a new technique for predicting the resource demand requirements of services implemented by multitier systems. Accurate demand estimates are essential to ensure the efficient provisioning of services in an increasingly service-oriented world. The demand estimation technique proposed in this paper has several advantages compared with regression-based demand estimation techniques, which many practitioners employ today. In contrast to regression, it does not suffer from the problem of multicollinearity, it provides more reliable aggregate resource demand and confidence interval predictions, and it offers a measurement-based validation test. The technique can be used to support system sizing and capacity planning exercises, costing and pricing exercises, and to predict the impact of changes to a service upon different service customers.

Predictive modelling of SAP ERP applications: challenges and solutions

Analytic performance models are being increasingly used to support system runtime optimization. This paper considers the modelling features needed to predict the response time behaviour of an industrial enterprise resource planning (ERP) application, SAP ERP. A number of studies have reported modelling success with the application of basic product-form Queueing Network Models (QNMs) to multi-tier systems. Such QNMs are often preferred in the context of optimization studies due to the low computational costs of their solution. However, we show that these simple models do not support many important features required to accurately characterize industrial applications such as ERP systems. Specifically, our results indicate that software threading levels, asynchronous database calls, priority scheduling, multiple phases of processing, and the parallelism offered by multi-core processors all have a significant impact on response time that cannot be neglected. Starting from these observations, the paper shows that Layered Queueing Models (LQMs) are a robust alternative to basic QNMs, while still enjoying analytical solution algorithms that facilitate their integration in optimization studies. A case study for a sales and distribution workload demonstrates that many of the features supported by LQMs are critical for achieving good prediction accuracy. Results show that, remarkably, all of the features we considered that are not captured by basic product-form QNMs are needed to predict mean response times to within 15% of measured values for a wide range of load levels. If any key feature is absent, the mean response time estimates could differ by 36% to 117% compared to the measured values, thus making the case that such non-product-form modelling features are needed for complex real-world applications.

Resource demand modeling for multi-tier services

We present a new technique for predicting the resource demand requirements of services implemented by multi-tier systems. Accurate demand estimates are essential to ensure the efficient provisioning of services in an increasingly service-oriented world. The demand estimation technique proposed in this paper has several advantages compared with regression-based demand estimation techniques, which many practitioners employ today. In contrast to regression, it does not suffer from the problem of multicollinearity, it provides more reliable aggregate resource demand and confidence interval predictions, and it offers a measurement-based validation test. The technique can be used to support system sizing and capacity planning exercises, costing and pricing exercises, and to predict the impact of changes to a service upon different service customers.

BAP: a benchmark-driven algebraic method for the performance engineering of customized services

This paper describes our joint research on performance engineering methods for services in shared resource utilities. The techniques support the automated sizing of a customized service instance and the automated creation of performance validation tests for the instance. The performance tests permit fine-grained control over inter-arrival time and service time burstiness to validate sizing and facilitate the development and validation of adaptation policies. Our novel research on sizing also takes into account the impact of workload factors that contribute to such burstiness. The methods are automated, integrated, and exploit an algebraic approach to workload modelling that relies on per-service benchmark suites with benchmarks that can be automatically executed within utilities. The benchmarks and their performance results are reused to support a Benchmark-driven Algebraic method for the Performance (BAP) engineering of customized services.

Simulation model driven performance evaluation for enterprise applications

Performance evaluations for enterprise applications running over IT systems are difficult to carry out given the multiplicity and variability of the operational components that constitute the dispersed IT infrastructures. To overcome this challenge, most of the approaches for performance assessment employ benchmarking strategies. While benchmarking methods provide exact indications on the performance capability of the measured facility, the results so obtained mostly apply to specific physical implementations considered in benchmark runs. The information provided by benchmark data thus restricts the ability to carry out meaningful performance analysis unless wide varieties of physical scenarios are generated for comparative studies. Given the logistical drawbacks associated with benchmarking techniques, we therefore propose a flexible model-based approach to determine quantitative performance for applications in IT systems by producing a range of performance models through the use of generic components that are easily assembled in simulation environments. Our approach initially considers a Tier 2 model framework whose components are derived from the SAP Sell-from-Stock application routine running on a multi-core processor server. The modelled framework is extensible enough to provide the definitions of resource consumptions patterns of different applications as well as the variety of server hardware systems. The simulations of our initial models developed so far generate results that are comparable to measurements obtained for scenarios in the low and moderate loading levels.

Characterization, monitoring and evaluation of operational performance trends on server processor hardware

Enterprise IT environments have seen a sharp growth in content use due to the popularity of on-demand data-intensive applications. In turn, the huge demand in content has spawned off major developments such as growth and distribution of computing nodes as well as the adoption of various implementation technologies. Given the complexity brought to the makeup of business computing environments in addressing the above-mentioned factors, the critical planning task of determining the appropriate infrastructure sizes for supporting firm Quality of Service (QoS) guarantees becomes a very challenging undertaking to fulfil. Benchmarking methods are widely employed in calibrating attainable performance in IT solutions, but these have the drawback of presenting output performance metrics as composite measurements that only give an end-to-end perspective. As an enhancement to benchmarking approaches, we explore the use of Performance Monitoring Counters (PMCs) in obtaining detailed operational performance of CPU and memory hardware. Performance Monitoring Counters (PMCs) are onchip registers found on most modern processor hardware. We use PMC-derived measurements to validate cache performance trends that have been derived analytically, and in the course of validations, PMC data is also used to investigate the nature and character of surges in cache miss events, which emerge as the memory load generated by runtime processes increases.

Characterization, monitoring and evaluation of operational performance trends on server processor hardware (abstracts only)

Enterprise IT environments have seen a sharp growth in content use due to the popularity of on-demand data-intensive applications. In turn, the huge demand in content has spawned off major developments such as growth and distribution of computing nodes as well as the adoption of various implementation technologies. Given the complexity brought to the makeup of business computing environments in addressing the above-mentioned factors, the critical planning task of determining the appropriate infrastructure sizes for supporting firm Quality of Service (QoS) guarantees becomes a very challenging undertaking to fulfil. Benchmarking methods are widely employed in calibrating attainable performance in IT solutions, but these have the drawback of presenting output performance metrics as composite measurements that only give an end-to-end perspective. As an enhancement to benchmarking approaches, we explore the use of Performance Monitoring Counters (PMCs) in obtaining detailed operational performance of CPU and memory hardware. Performance Monitoring Counters (PMCs) are onchip registers found on most modern processor hardware. We use PMC-derived measurements to validate cache performance trends that have been derived analytically, and in the course of validations, PMC data is also used to investigate the nature and character of surges in cache miss events, which emerge as the memory load generated by runtime processes increases.