Narendran Sachindran

Problem Determination in Enterprise Middleware Systems using Change Point Correlation of Time Series Data

Clustered enterprise middleware systems employing dynamic workload scheduling are susceptible to a variety of application malfunctions that can manifest themselves in a counterintuitive fashion and cause debilitating damage. Until now, diagnosing problems in that domain involves investigating log files and configuration settings and requires in-depth knowledge of the middleware architecture and application design. This paper presents a method for problem determination using change point detection techniques and problem signatures consisting of a combination of changes (or absence of changes) in different metrics. We implemented this approach on a clustered middleware system and applied it to the detection of the storm drain condition: a debilitating problem encountered in clustered systems with counterintuitive symptoms. Our experimental results show that the system detects 93% of storm drain faults with no false positives

Fast extraction of adaptive change point based patterns for problem resolution in enterprise systems

Enterprise middleware systems typically consist of a large cluster of machines with stringent performance requirements. Hence, when a performance problem occurs in such environments, it is critical that the health monitoring software identifies the root cause with minimal delay. A technique commonly used for isolating root causes is rule definition, which involves specifying combinations of events that cause particular problems. However, such predefined rules (or problem signatures) tend to be inflexible, and crucially depend on domain experts for their definition. We present in this paper a method that automatically generates change point based problem signatures using administrator feedback, thereby removing the dependence on domain experts. The problem signatures generated by our method are flexible, in that they do not require exact matches for triggering, and adapt as more information becomes available. Unlike traditional data mining techniques, where one requires a large number of problem instances to extract meaningful patterns, our method requires few fault instances to learn problem signatures. We demonstrate the efficacy of our approach by learning problem signatures for five common problems that occur in enterprise systems and reliably recognizing these problems with a small number of learning instances.

SASH: Enabling continuous incremental analytic workflows on Hadoop

There is an emerging class of enterprise applications in areas such as log data analysis, information discovery, and social media marketing that involve analytics over large volumes of unstructured and semi-structured data. These applications are leveraging new analytics platforms based on the MapReduce framework and its open source Hadoop implementation. While this trend has engendered work on high-level data analysis languages, NoSQL data stores, workflow engines etc., there has been very little attention to the challenges of deploying analytic workflows into production for continuous operation. In this paper, we argue that an essential platform component for enabling continuous production analytic workflows is an analytics store. We highlight five key requirements that impact the design of such a store: (i) efficient incremental operations, (ii) flexible storage model for hierarchical data, (iii) snapshot support (iv) object-level incremental updates, and (v) support for handling change sets. We describe the design of SASH, a scalable analytics store that we have developed on top of HBase to address these requirements. Using the workload from a production workflow that powers search within IBMs intranet and extranet, we demonstrate orders of magnitude improvement in IO performance using SASH.

A framework for migrating production snapshots of composite applications to virtualized environments

Migrating production applications from physical datacenters to a virtualized environment is becoming essential to reduce operational costs. In order to avoid direct access to production systems, migration from disk snapshots is preferable. Model based migration approaches are not suitable for this purpose since they need to perform configuration discovery on production systems. Recent approaches that rely upon an isolated network require special setup and are limited to handling network configurations only. In this paper we present a framework for instantiating an application from disk snapshots in a virtualized environment. Our approach comprises of formally specifying the knowledge about product configurations and compiling the knowledge into a plan. The plan is capable of discovering application configurations in an inconsistent configuration setup. A key feature of our framework is that knowledge is specified once per product and re-used across applications that use the product. We have implemented a prototype of our approach and evaluated it on a real world application. The evaluation demonstrates the feasibility of our approach for migration using disk snapshots.

Real-time problem localization for synchronous transactions in HTTP-based composite enterprise applications

Loosely-coupled composite enterprise applications based on modern Web technologies are becoming increasingly popular. While composing such applications is appealing for a number of reasons, the distributed nature of the applications makes problem determination difficult. Stringent service level agreements in these environments require rapid localization of failing and poorly performing services. We present in this paper a method that performs real-time transaction level problem determination by tracking synchronous transaction flows in HTTP based composite enterprise applications. Our method relies on instrumentation of service requests and responses to transmit downstream path and monitoring information in realtime. Further, our method applies change-point based techniques on monitored information at the point of origin of a transaction, and quickly detects anomalies in the performance of invoked services. Since our method performs transaction level monitoring, it avoids the pitfalls associated with techniques that use aggregate performance metrics. Additionally, since we use change-point based techniques to detect problems, our method is more robust than error-prone static threshold based techniques.