Sandeep M. Uttamchandani

An Empirical Exploration of Black-Box Performance Models for Storage Systems

The effectiveness of automatic storage management depends on the accuracy of the storage performance models that are used for making resource allocation decisions. Several approaches have been proposed for modeling. Black-box approaches are the most promising in real-world storage systems because they require minimal device specific information, and are self-evolving with respect to changes in the system. However, blackbox techniques have been traditionally considered inaccurate and non-converging in real-world systems. This paper evaluates a popular off-the-shelf black-box technique for modeling a real-world storage environment. We measured the accuracy of performance predictions in single workload and multiple workload environments. We also analyzed accuracy of different performance metrics namely throughput, latency, and detection of saturation state. By empirically exploring improvements for the model accuracy, we discovered that by limiting the component model training for the nonsaturated zone only and by taking into account the number of outstanding IO requests, the error rate of the throughput model is 4.5% and the latency model is 19.3%. We also discovered that for systems with multiple workloads, it is necessary to consider access characteristics of each workload as input parameters for the model. Lastly, we report results on the sensitivity of model accuracy as a function of the amount of bootstrapping data.

Genesis: A Scalable Self-Evolving Performance Management Framework for Storage Systems

data-center environment, the administrator needs to understand the root-cause of the issue. The growing trend of system virtualization, combined with the need to support end-to-end performance goals for enterprise applications, have made root-cause analysis a nontrivial problem - administrators are required to manually parse all hardware events, configuration modifications, and changes in access characteristics, across all tiers of the IO path from application servers to the disks. We propose a framework that assists storage administrators with root-cause analysis in distributed systems. GENESIS consists of three key modules: Abnormality Detection, Snapshot Generation, and Diagnosis. The Abnormality Detection module uses clustering algorithms to create and constantly evolve the normality models of measurable parameters in components. The Snapshot Generator is triggered by a combination of abnormality detection and policies to take compact snapshots of the system state for analysis whenever a significant change occurs. The Diagnosis module parses the snapshots and shortlists the root-cause for the administrator using knowledge about the impact of the run-time changes on IO performance. We have implemented an initial proof-of-concept of GENESIS in GPFS (a high performance distributed file-system) and validated its operation for several interesting real-world scenarios. Encouraged by the results, we are currently deploying our prototype in an existing data center environment.

Evolution of storage management: transforming raw data into information

Exponential growth in storage requirements and an increasing number of heterogeneous devices and application policies are making enterprise storage management a nightmare for administrators. Back-of-the-envelope calculations, rules of thumb, and manual correlation of individual device data are too error prone for the day-to-day administrative tasks of resource provisioning, problem determination, performance management, and impact analysis. Storage management tools have evolved over the past several years from standardizing the data reported by storage subsystems to providing intelligent planners. In this paper, we describe that evolution in the context of the IBM Total Storage® Productivity Center (TPC)--a suite of tools to assist administrators in the day-to-day tasks of monitoring, configuring, provisioning, managing change, analyzing configuration, managing performance, and determining problems. We describe our ongoing research to develop ways to simplify and automate these tasks by applying advanced analytics on the performance statistics and raw configuration and event data collected by TPC using the popular Storage Management Initiative-Specification (SMI-S). In addition, we provide details of SMART (storage management analytics and reasoning technology) as a library that provides a collection of data-aggregation functions and optimization algorithms.

ChargeView: An integrated tool for implementing chargeback in IT systems

Most organizations are becoming increasingly reliant on IT product and services to manage their daily operations. The total cost of ownership (TCO), which includes the hardware and software purchase cost, management cost, etc., has significantly increased and forms one of the major portions of the total expenditure of the company. CIOs have been struggling to justify the increased costs and at the same time fulfill the IT needs of their organizations. For businesses to be successful, these costs need to be carefully accounted and attributed to specific processes or user groups/departments responsible for the consumption of IT resources. This process is called IT chargeback and although desirable, is hard to implement because of the increased consolidation of IT resources via technologies like virtualization. Current IT chargeback methods are either too complex or too adhoc, and often a times lead to unnecessary tensions between IT and business departments and fail to achieve the goal for which chargeback was implemented. This paper presents a new tool called ChargeView that automates the process of IT costing and chargeback. First, it provides a flexible hierarchical framework that encapsulates the cost of IT operations at different level of granularity. Second, it provides an easy way to account for different kind of hardware and management costs. Third, it permits implementation of multiple chargeback policies that fit the organization goals and establishes relationship between the cost and the usage by different users and departments within an organization. Finally, its advanced analytics functions can keep track of usage and cost trends, measure unused resources and aid in determining service pricing. We discuss the prototype implementation of ChargeView and show how it has been used for managing complex systems and storage networks.

Automated planners for storage provisioning and disaster recovery

Introducing an application into a data center involves complex interrelated decision-making for the placement of data (where to store it) and resiliency in the event of a disaster (how to protect it). Automated planners can assist administrators in making intelligent placement and resiliency decisions when provisioning for both new and existing applications. Such planners take advantage of recent improvements in storage resource management and provide guided recommendations based on monitored performance data and storage models. For example, the IBM Provisioning Planner provides intelligent decision-making for the steps involved in allocating and assigning storage for workloads. It involves planning for the number, size, and location of volumes on the basis of workload performance requirements and hierarchical constraints, planning for the appropriate number of paths, and enabling access to volumes using zoning, masking, and mapping. The IBM Disaster Recovery (DR) Planner enables administrators to choose and deploy appropriate replication technologies spanning servers, the network, and storage volumes to provide resiliency to the provisioned application. The DR Planner begins with a list of high-level application DR requirements and creates an integrated plan that is optimized on criteria such as cost and solution homogeneity. The Planner deploys the selected plan using orchestrators that are responsible for failover and failback.

DecisionQoS: an adaptive, self-evolving QoS arbitration module for storage systems

As a consequence of the current trend towards consolidating computing, storage and networking infrastructures into large centralized data centers, applications compete for shared resources. Open enterprise systems are not designed to provide performance guarantees in the presence of sharing; unregulated competition is very likely to result in a free-for-all where some applications monopolize resources while others starve. Rule-based solutions to the resource arbitration problem suffer from excessive complexity, brittleness, and limitations in their expressive power. We present DecisionQoS, a novel approach for arbitrating resources among multiple competing clients while enforcing QoS guarantees. DecisionQoS requires system administrators to provide a minimal, declarative amount of information about the system and the workloads running on it. That initial input is continuously refined and augmented at run time, by monitoring the systems performance and its reaction to resource allocation decisions. When faced with incomplete information, or with changes in the workload requirements or system capabilities, DecisionQoS adapts to them by applying machine learning techniques; the resulting scheme is highly resilient to unforeseen events. Moreover, it overcomes significant shortcomings of pre-existing, rule-based policy management systems.

Modeling the Fault Tolerance Consequences of Deduplication

Modern storage systems are employing data deduplication with increasing frequency. Often the storage systems on which these techniques are deployed contain important data, and utilize fault-tolerant hardware and software to improve the reliability of the system and reduce data loss. We suggest that data deduplication introduces inter-file relationships that may have a negative impact on the fault tolerance of such systems by creating dependencies that can increase the severity of data loss events. We present a framework composed of data analysis methods and a model of data deduplication that is useful in studying the reliability impact of data deduplication. The framework is useful for determining a deduplication strategy that is estimated to satisfy a set of reliability constraints supplied by a user.

Eos: An Approach of Using Behavior Implications for Policy-Based Self-Management

Systems are becoming exceedingly complex to manage. As such, there is an increasing trend towards developing systems that are self-managing. Policy-based infrastructures have been used to provide a limited degree of automation, by associating actions to system-events. In the context of self-managing systems, the existing policy-specification model fails to capture the following: a) The impact of a rule on system behavior (behavior implications). This is required for automated decision-making. b) Learning mechanisms for refining the invocation heuristics by monitoring the impact of rules.

BRAHMA: Planning Tool for Providing Storage Management as a Service

Storage management is becoming the largest component in the overall cost of storage ownership. Most organizations are trying to either consolidate their storage management operations or outsource them to a storage service provider (SSP) in order to contain the management costs. Currently, there do not exist any planning tools that help the clients and the SSPs in figuring out the best outsourcing option. In this paper we present a planning tool, Brahma, that specifically addresses the above mentioned problem, as Brahma is capable of providing solutions where the management tasks are split between the client and SSP at a finer granularity. Our tool is unique because: (a) in addition to hardware/software resources, it also takes human skill set as an input; (b) it takes planning time window as input because plans that are optimal for a given time period (e.g. a month) might not necessarily be the most optimum for a different time period (e.g. a year); (c) it can be used separately by both the client and the SSP to do their respective planning; (d) it allows the client and the SSP to propose alternative solutions if certain input service level agreements can be relaxed. We have implemented BRAHMA, and our experiment results show that there definitely are cost benefits that one can attain by having a tool with the above mentioned functional properties.

End-to-end disaster recovery planning: From art to science

We present the design and implementation of ENDEAVOUR - a framework for integrated end-to-end disaster recovery (DR) planning. Unlike existing research that provides DR planning within a single layer of the IT stack (e.g. storage controller based replication), ENDEAVOUR can choose technologies and composition of technologies across multiple layers like virtual machines, databases and storage controllers. ENDEAVOUR uses a canonical model of available replication technologies at all layers, explores strategies to compose them, and performs a novel map-search-reduce heuristic to identify the best DR plans for given administrator requirements. We present a detailed analysis of ENDEAVOUR including empirical characterization of various DR technologies, their composition, and a end-to-end case study.