Rohit Wagle

Towards Autonomic Fault Recovery in System-S

System-S is a stream processing infrastructure which enables program fragments to be distributed and connected to form complex applications. There may be potentially tens of thousands of interdependent and heterogeneous program fragments running across thousands of nodes. While the scale and interconnection imply the need for automation to manage the program fragments, the need is intensified because the applications operate on live streaming data and thus need to be highly available. System-S has been designed with components that autonomically manage the program fragments, but the system components themselves are also susceptible to failures which can jeopardize the system and its applications. The work we present addresses the self healing nature of these management components in System-S. In particular, we show how one key component of System-S, the job management orchestrator, can be abruptly terminated and then recover without interrupting any of the running program fragments by reconciling with other autonomous system components. We also describe techniques that we have developed to validate that the system is able to autonomically respond to a wide variety of error conditions including the abrupt termination and recovery of key system components. Finally, we show the performance of the job management orchestrator recovery for a variety of workloads.

SODA: an optimizing scheduler for large-scale stream-based distributed computer systems

This paper describes the SODA scheduler for System S, a highly scalable distributed stream processing system. Unlike traditional batch applications, streaming applications are open-ended. The system cannot typically delay the processing of the data. The scheduler must be able to shift resource allocation dynamically in response to changes to resource availability, job arrivals and departures, incoming data rates and so on. The design assumptions of System S, in particular, pose additional scheduling challenges. SODA must deal with a highly complex optimization problem, which must be solved in real-time while maintaining scalability. SODA relies on a careful problem decomposition, and intelligent use of both heuristic and exact algorithms. We describe the design and functionality of SODA, outline the mathematical components, and describe experiments to show the performance of the scheduler.

Constructing consumer profiles from social media data

Social media is playing a growing role in providing consumer feedback to companies about their products and services. To maximize the benefit of this feedback, companies want to know how different consumer-segments they are interested in, such as parents, frequent travelers, and comic book fans react to their products and campaigns. In this paper, we describe how constructing consumer profiles is valuable to obtain such insights. We present the challenges in analyzing noisy social media data and the techniques we employ for building the profiles. We also present detailed experimental results from the analysis of over seven billion messages to construct profiles of over 100 million consumers. We demonstrate how consumer profiles can help in understanding consumer feedback by different key segments using a TV show analysis scenario.

Surfacing time-critical insights from social media

We propose to demonstrate an end-to-end framework for leveraging time-sensitive and critical social media information for businesses. More specifically, we focus on identifying, structuring, integrating, and exposing timely insights that are essential to marketing services and monitoring reputation over social media. Our system includes components for information extraction from text, entity resolution and integration, analytics, and a user interface.

Building user-defined runtime adaptation routines for stream processing applications

Stream processing applications are deployed as continuous queries that run from the time of their submission until their cancellation. This deployment mode limits developers who need their applications to perform runtime adaptation, such as algorithmic adjustments, incremental job deployment, and application-specific failure recovery. Currently, developers do runtime adaptation by using external scripts and/or by inserting operators into the stream processing graph that are unrelated to the data processing logic. In this paper, we describe a component called orchestrator that allows users to write routines for automatically adapting the application to runtime conditions. Developers build an orchestrator by registering and handling events as well as specifying actuations. Events can be generated due to changes in the system state (e.g., application component failures), built-in system metrics (e.g., throughput of a connection), or custom application metrics (e.g., quality score). Once the orchestrator receives an event, users can take adaptation actions by using the orchestrator actuation APIs. We demonstrate the use of the orchestrator in IBMs System S in the context of three different applications, illustrating application adaptation to changes on the incoming data distribution, to application failures, and on-demand dynamic composition.

Distributed middleware reliability and fault tolerance support in system S

We describe a fault-tolerance technique for implementing operations in a large-scale distributed system that ensures that all the components will eventually have a consistent view of the system even in the face of component failures. To achieve this, we break the distributed operation into a series of smaller operations, each of which is local to a single component, carefully linked together. Thus, the effect of a component failure and restart in the middle of a multi-component operation is limited to that component and its immediate neighbors. This framework is used in System S, a commercial grade stream processing platform. In that context we will show empirically that our approach is effective and imposes low overhead on distributed inter-component operations.

VLAN-Based Routing Infrastructure for an All-Optical Circuit Switched LAN

Exploring the use of all-optical MEM switches on Local Area Network (LAN) to increase bandwidth and reduce energy cost has received increasing attention. Compared to traditional electronic switches, an all-optical switch provides higher bandwidth, less energy cost and cheaper wiring. Once the optical port count per switch becomes a resource constraint, an all-optical switched core can change the network topology dynamically to redistribute bandwidth resources between computing hosts. This feature is particularly useful for stream processing systems whose communication patterns vary over time and where rebalancing of networking resource is needed periodically. The work we present in this paper is a practical solution for high level software systems to route through a reconfigurable optical MEM switched LAN. Our solution can be readily applied on commercial switches with standard Layer-2 protocols. Network reconfiguration time and round-trip delay are measured. Our implementation is validated with the IBM System S stream processing system.

The best of two worlds: Integrating IBM InfoSphere Streams with Apache YARN

The seamless confluence of data in motion and data at rest has the potential to redefine the Big Data analytics landscape in a diverse range of domains. To make this happen, existing data intensive computing frameworks need to be repurposed and integrated at control, data, and management levels. Towards this end, we present the system level integration of IBM InfoSphere Streams with Apache YARN. Our design leverages the key differentiating features of the two frameworks to blend high throughput batch-processing with near line-rate, low latency stream-processing. In addition, both frameworks are able to share resources and offer the same interfaces that their users are accustomed to. Using two real-world examples, we illustrate how such a system can be used in production.

The X-flex cross-platform scheduler: who's the fairest of them all?

We introduce the X-Flex cross-platform scheduler. X-Flex is intended as an alternative to the Dominant Resource Fairness (DRF) scheduler currently employed by both YARN and Mesos. There are multiple design differences between X-Flex and DRF. For one thing, DRF is based on an instantaneous notion of fairness, while X-Flex monitors instantaneous fairness in order to take a long-term view. The definition of instantaneous fairness itself is different among the two schedulers. Furthermore, the packing of containers into processing nodes in DRF is done online, while in X-Flex it is performed offline in order to improve packing quality. Finally, DRF is essentially an extension to multiple dimensions of the Fair MapReduce scheduler. As such it makes scheduling decisions at a very low level. X-Flex, on the other hand, takes the perspective that some frameworks have sufficient structure to make higher level scheduling decisions. So X-Flex allows this, and also gives platforms a great deal of autonomy over the degree of sharing they will permit with other platforms. We describe the technical details of X-Flex and provide experiments to show its excellent performance.

An evaluation of zookeeper for high availability in system S

ZooKeeper provides scalable, highly available coordination services for distributed applications. In this paper, we evaluate the use of ZooKeeper in a distributed stream computing system called System S to provide a resilient name service, dynamic configuration management, and system state management. The evaluation shed light on the advantages of using ZooKeeper in these contexts as well as its limitations. We also describe design changes we made to handle named objects in System S to overcome the limitations. We present detailed experimental results, which we believe will be beneficial to the community.