Sujay Parekh

Using control theory to achieve service level objectives in performance management

A widely used approach to achieving service level objectives for a software system (e.g., an email server) is to add a controller that manipulates the target systems tuning parameters. We describe a methodology for designing such controllers for software systems that builds on classical control theory. The classical approach proceeds in two steps: system identification and controller design. In system identification, we construct mathematical models of the target system. Traditionally, this has been based on a first-principles approach, using detailed knowledge of the target system. Such models can be complex and difficult to build, validate, use, and maintain. In our methodology, a statistical (ARMA) model is fit to historical measurements of the target being controlled. These models are easier to obtain and use and allow us to apply control-theoretic design techniques to a larger class of systems. When applied to a Lotus Notes groupware server, we obtain model-fits with R2 no lower than 75% and as high as 98%. In controller design, an analysis of the models leads to a controller that will achieve the service level objectives. We report on an analysis of a closed-loop system using an integral control law with Lotus Notes as the target. The objective is to maintain a reference queue length. Using root-locus analysis from control theory, we are able to predict the occurrence (or absence) of controller-induced oscillations in the systems response. Such oscillations are undesirable since they increase variability, thereby resulting in a failure to meet the service level objective. We implement this controller for a real Lotus Notes system, and observe a remarkable correspondence between the behavior of the real system and the predictions of the analysis. This indicates that the control theoretic analysis is sufficient to select controller parameters that meet the desired goals, and the need for simulations is reduced.

MIMO control of an Apache web server: modeling and controller design

This paper considers the efficacy of feedback control in improving the performance of computing systems. Computing systems typically have many competing performance goals which are affected by several external variables. A feedback control strategy is desirable because well established techniques exist to handle these performance trade-offs and external disturbances. In order to employ such a strategy, decisions need to be made about the inputs, outputs, sample time, model type, and performance measures. This paper describes this process, which is often nebulous for computing systems, in the context of an Apache web server. A linear multi-input multi-output model of the system is identified experimentally and used to design several feedback controllers. Experimental results are presented showing the problems associated with a pure pole placement design and effectiveness of LQ control based techniques. The paper concludes with a discussion of future work.

A control theory foundation for self-managing computing systems

The high cost of operating large computing installations has motivated a broad interest in reducing the need for human intervention by making systems self-managing. This paper explores the extent to which control theory can provide an architectural and analytic foundation for building self-managing systems. Control theory provides a rich set of methodologies for building automated self-diagnosis and self-repairing systems with properties such as stability, short settling times, and accurate regulation. However, there are challenges in applying control theory to computing systems, such as developing effective resource models, handling sensor delays, and addressing lead times in effector actions. We propose a deployable testbed for autonomic computing (DTAC) that we believe will reduce the barriers to addressing research problems in applying control theory to computing systems. The initial DTAC architecture is described along with several problems that it can be used to investigate.

Online response time optimization of Apache web server

Properly optimizing the setting of configuration parameters can greatly improve performance, especially in the presence of changing workloads. This paper explores approaches to online optimization of the Apache web server, focusing on the MaxClients parameter (which controls the maximum number of workers). Using both empirical and analytic techniques, we show that MaxClients has a concave upward effect on response time and hence hill climbing techniques can be used to find the optimal value of MaxClients. We investigate two optimizers that employ hill climbing--one based on Newtons Method and the second based on fuzzy control. A third technique is a heuristic that exploits relationships between bottleneck utilizations and response time minimization. In all cases, online optimization reduces response times by a factor of 10 or more compared to using a static, default value. The trade-offs between the online schemes are as follows. Newtons method is well known but does not produce consistent results for highly variable data such as response times. Fuzzy control is more robust, but converges slowly. The heuristic works well in our prototype system, but it may be difficult to generalize because it requires knowledge of bottleneck resources and an ability to measure their utilizations.

Managing Web server performance with AutoTune agents

Managing the performance of e-commerce sites is challenging. Site content changes frequently, as do customer interests and business plans, contributing to dynamically varying workloads. To maintain good performance, system administrators must tune their information technology environment on an ongoing basis. Unfortunately, doing so requires considerable expertise and increases the total cost of system ownership. In this paper, we propose an agent-based solution that not only automates the ongoing system tuning but also automatically designs an appropriate tuning mechanism for the target system. We illustrate this in the context of managing a Web server. There we study the problem of controlling CPU and memory utilization of an Apache® Web server using the application-level tuning parameters MaxClients and KeepAlive, which are exposed by the server. Using the AutoTune agent framework under the Agent Building and Learning Environment (ABLE), we construct agents to fully automate a control-theoretic methodology that involves model building, controller design, and run-time feedback control. Specifically, we design (1) a modeling agent that builds a dynamic system model from the controlled server run data, (2) a controller design agent that uses optimal control theory to derive a feedback control algorithm customized to that server, and (3) a run-time control agent that deploys the feedback control algorithm in an on-line real- time environment to automatically manage the Web server. The designed autonomic feedback control system is able to handle the dynamic and interrelated dependencies between the1 tuning parameters and the performance metrics with guaranteed stability from control theory. The effectiveness of the AutoTune agents is demonstrated through experiments involving variations in workload, server capacity, and business objectives. The results also serve as a validation of the ABLE toolkit and the AutoTune agent framework.

Self-managing systems: a control theory foundation

The high cost of operating large computing installations has motivated a broad interest in reducing the need for human intervention by making systems self-managing. This paper explores the extent to which control theory can provide an architectural and analytic foundation for building self-managing systems, either from new components or layering on top of existing components. Further, we propose a deployable testbed for autonomic computing (DTAC) that we believe will reduce the barriers to addressing key research problems in autonomic computing. The initial DTAC architecture is described along with several problems that it can be used to investigate.

Controlling Quality of Service in Multi-Tier Web Applications

The need for service differentiation in Internet services has motivated interest in controlling multi-tier web applications. This paper describes a tier-to-tier (T2T) management architecture that supports decentralized actuator management in multi-tier systems, and a testbed implementation of this architecture using commercial software products. Based on testbed experiments and analytic models, we gain insight into the value of coordinated exploitation of actuators on multiple tiers, especially considerations for control efficiency and control granularity. For control efficiency, we show that more effective utilization of tiers can be achieved by using actuators on the bottleneck tier rather than only using actuators on the entry tier. For granularity of control (the ability to achieve a wide range of service level objectives) we show that a fine granularity of control can be achieved through a coordinated, cross-tier exploitation of coarse grained actuators (e.g., multiprogramming level), an approach that can greatly reduce controllerinduced variability.

Feedback control of a Lotus Notes server: modeling and control design

This paper considers the modeling and feedback control of a mail server running Lotus Notes/sup TM/. Computing systems such as this typically have two competing control goals: maximize throughput and minimize response time. To achieve these goals, a control input (tuning control) is used to limit the number of users allowed to connect to the mail server at any one time. The measured output is the server queue length-the number of requests that are waiting to be processed. Because response time is a client metric and thus difficult to measure at the server, we formulate the control problem as tracking a reference server queue length. A linear input-output model of the system is identified experimentally and used to design an integral controller. Losses in the queue length measurement due to the fact that requests are only logged after they are served are accounted for by another linear model. Experimental results are presented showing the effectiveness of a low-gain controller and the saturation problems experienced by a high-gain controller. The paper concludes with a discussion of future work.

Incorporating cost of control into the design of a load balancing controller

Load balancing is widely used in computing systems as a way to optimize performance by reducing bottleneck utilizations, such as adjusting the size of buffer pools to balance resource demands in a database management system. Load balancing is generally approached as a constrained optimization problem in which only the benefits of load balancing are considered. However, the costs of control are important as well. Herein, we study the value of including in controller design the trade-off between the cost of transient imbalances in resource utilizations and the cost of changing resource allocations. An example of the latter are actions such as resizing buffer pools that can reduce throughputs. This is because requests for data in pools whose memory is reduced immediately have longer access times whereas requests for data in pools whose memory is increased must fill this memory with data from disk before accessed times are reduced. We frame our study of control costs in terms of the widely used linear quadratic regulator (LQR). We develop a cost model that allows us to specify the LQR Q and R matrices based on the impact on system performance of changing resource allocations and transient load imbalances. Our studies of a DB2 universal database server using benchmarks for online transaction processing and decision support workloads show that incorporating our cost model into the MIMO LQR controller results in a 14% improvement in performance beyond that achieved by dynamically allocating the size of buffers without properly considering the cost of control.

Managing the Performance Impact of Administrative Utilities

Administrative utilities (e.g., filesystem and database backups, garbage collection in the Java Virtual Machines) are an essential part of the operation of production systems. Since production work can be severely degraded by the execution of such utilities, it is desirable to have policies of the form “There should be no more than an x% degradation of production work due to utility execution.” Two challenges arise in providing such policies: (1) providing an effective mechanism for throttling the resource consumption of utilities and (2) continuously translating from policy expressions of “degradation units” into the appropriate settings for the throttling mechanism. We address (1) by using self-imposed sleep, a technique that forces utilities to slow down their processing by a configurable amount. We address (2) by employing an online estimation scheme in combination with a feedback loop. This throttling system is autonomous and adaptive and allows the system to self-manage its utilities to limit their performance impact, with only high-level policy input from the administrator. We demonstrate the effectiveness of these approaches in a prototype system that incorporates these capabilities into IBM’s DB2 Universal Database server.