Kavitha Ranganathan

Decoupling computation and data scheduling in distributed data-intensive applications

In high-energy physics, bioinformatics, and other disciplines, we encounter applications involving numerous, loosely coupled jobs that both access and generate large data sets. So-called Data Grids seek to harness geographically distributed resources for such large-scale data-intensive problems. Yet effective scheduling in such environments is challenging, due to a need to address a variety of metrics and constraints while dealing with multiple, potentially independent sources of jobs and a large number of storage, compute, and network resources. We describe a scheduling framework that addresses these problems. Within this framework, data movement operations may be either tightly bound to job scheduling decisions or, alternatively, performed by a decoupled, asynchronous process on the basis of observed data access patterns and load. We develop a family of algorithms and use simulation studies to evaluate various combinations. Our results suggest that while it is necessary to consider the impact of replication, it is not always necessary to couple data movement and computation scheduling. Instead, these two activities can be addressed separately, thus significantly simplifying the design and implementation.

Improving Data Availability through Dynamic Model-Driven Replication in Large Peer-to-Peer Communities

Efficient data sharing in global peer-to-peer systems is complicated by erratic node failure, unreliable network connectivity and limited bandwidth. Replicating data on multiple nodes can improve availability and response time. Yet determining when and where to replicate data in order to meet performance goals in large-scale systems with many users and files, dynamic network characteristics, and changing user behavior is difficult. We propose an approach in which peers create replicas automatically in a decentralized fashion, as required to meet availability goals. The aim of our framework is to maintain a threshold level of availability at all times. We identify a set of factors that hinder data availability and propose a model that decides when more replication is necessary. We evaluate the accuracy and performance of the proposed model using simulations. Our preliminary results show that the model is effective in predicting the required number of replicas in the system.

Simulation Studies of Computation and Data Scheduling Algorithms for Data Grids

Data Grids seek to harness geographically distributed resources for large-scale data-intensive problems. Such problems, involving loosely coupled jobs and large data-sets, are found in fields like high-energy physics, astronomy and bioinformatics. A variety of factors need to be considered for effective scheduling of resources in such environments: e.g., resource utilization, response time, global and local allocation policies and scalability. We propose a general and extensible scheduling architecture that addresses these issues. Within this architecture we develop a suite of job scheduling and data replication algorithms that we evaluate using simulations for a wide range of parameters. Our results show that it is important to evaluate the combined effectiveness of replication and scheduling strategies, rather than study them separately. More specifically, we find that scheduling jobs to locations that contain the data they need and asynchronously replicating popular data-sets to remote sites, works rather well.

Incentive mechanisms for large collaborative resource sharing

We study the nature of sharing resources in distributed collaborations such as Grids and peer-to-peer systems. By applying the theoretical framework of the multi-person prisoners dilemma to this resource sharing problem, we show that in the absence of incentive schemes, individual users are apt to hold back resources, leading to decreased system utility. Using both the theoretical framework as well as simulations, we compare and contrast three different incentive schemes aimed at encouraging users to contribute resources. Our results show that soft-incentive schemes are effective in incentivizing autonomous entities to collaborate, leading to increased gains for all participants in the system.

Evolving toward the perfect schedule: co-scheduling job assignments and data replication in wide-area systems using a genetic algorithm

Traditional job schedulers for grid or cluster systems are responsible for assigning incoming jobs to compute nodes in such a way that some evaluative condition is met. Such systems generally take into consideration the availability of compute cycles, queue lengths, and expected job execution times, but they typically do not account directly for data staging and thus miss significant associated opportunities for optimisation. Intuitively, a tighter integration of job scheduling and automated data replication can yield significant advantages due to the potential for optimised, faster access to data and decreased overall execution time. In this paper we consider data placement as a first-class citizen in scheduling and use an optimisation heuristic for generating schedules. We make the following two contributions. First, we identify the necessity for co-scheduling job dispatching and data replication assignments and posit that simultaneously scheduling both is critical for achieving good makespans. Second, we show that deploying a genetic search algorithm to solve the optimal allocation problem has the potential to achieve significant speed-up results versus traditional allocation mechanisms. Through simulation, we show that our algorithm provides on average an approximately 20-45% faster makespan than greedy schedulers.

E-governance for development: a focus on rural India

This book on e-governance for development focuses more on “the governance and development aspects rather than the e.” This, as the author clarifies, is not accidental but rather an attempt to illustrate and bring to focus the social context that plays a major role in the success of any kind of innovation. Going by the title of the book and its setting in the Indian context, it may appear that the book is about the National eGovernance Program (NeGP) and its impact on human development in India. The vision of NeGP is that all Government services should become accessible to common citizens in their locality through a one-stop-shop ensuring efficiency, transparency and reliability at affordable costs. Early experience indicates a focus on regulatory services, rather than the services that can have a high developmental impact. However, the author has used a broad definition of e-governance to include ICT-enabled back-end support activities for planning and monitoring of developmental programs. In that respect, the book will be useful for an audience who is interested in the broader topic of ICT for development and not just e-governance. The first half of the book contains a comprehensive literature review on development and governance, and linkages between the two. The first few chapters quickly acquaint the reader with the history of governance and development research in India. The book then goes on to discuss e-governance in the Indian context. While this half of the book made for an interesting read, we were eagerly waiting for what we considered the meat of the book – the three in-depth case studies on rural e-governance projects, from three different states in India. The first is the study of MIS for SHG (self-help group) projects in Gujarat, the second a study of the Akshaya telecenter project in Kerala and the third case analyzes the reporting system for healthcare projects in rural Karnataka. Each case is from a different sector and while the Akshaya project deals with e-services, the other two are e-administrative projects. It is obvious that the author has spent considerable time in the field studying the three projects – they are detailed and reflect findings at the grassroots level, captured over a span of several months. The case studies reveal important learnings – such as the significance of the role of the state to ensure that the social goals of a project are met along with the economic objectives, and the importance of the field worker to ensure that the project remains locally relevant. While these leanings are significant, many of them apply to any development project and are not specific to an e-governance project. This may not necessarily be a limitation but somewhere along the way, we felt a little short-changed as we were looking for the “e” aspect to have played a greater role in this book – given that the title of the book promised e-governance. The book has a strong focus on exploring the role played by society for successful project implementation and states that “improving systems of governance is a social and not a technological activity.” While the importance of social factors in the success of any ICT4D project cannot be overemphasized, surely the role of technology cannot be disregarded. It is fairly well established by now that considering only the “tech” aspect while ignoring the social aspects in an ICT4D project spells disaster, but the book could have perhaps made an important contribution by elucidating why this happens in most ICT4D projects. The book tries to bridge multiple disciplines and this is a difficult path where few have succeeded. It provides a refreshing perspective on how to evaluate a development project and

Computation scheduling and data replication algorithms for data Grids

Data Grids seek to harness geographically distributed resources for large-scale data-intensive problems such as those encountered in high energy physics, bioinformatics, and other disciplines. These problems typically involve numerous, loosely coupled jobs that both access and generate large data sets. Effective scheduling in such environments is challenging, because of a need to address a variety of metrics and constraints (e.g., resource utilization, response time, global and local allocation policies) while dealing with multiple, potentially independent sources of jobs and a large number of storage, compute, and network resources.We describe a scheduling framework that addresses these problems. Within this framework, data movement operations may be either tightly bound to job scheduling decisions or performed by a decoupled, asynchronous process on the basis of observed data access patterns and load. We develop a family of job scheduling and data movement (replication) algorithms and use simulation studies to evaluate various combinations. Our results suggest that while it is necessary to consider the impact of replication on the scheduling strategy, it is not always necessary to couple data movement and computation scheduling. Instead, these two activities can be addressed separately, thus significantly simplifying the design and implementation of the overall Data Grid system.

A Voice for the Voiceless: Peer-to-peer Mobile Phone Networks for a Community Radio Service

We propose a new application for mobile ad hoc networks (MANETs) – community radio. We argue how MANETS help overcome important limitations in how community radio is currently operationalized. We identify critical design elements for a MANET based community radio service and propose a broad architecture for the same. We then investigate a most critical issue – the choice of the network wide broadcast protocol for the audio content. We identify desired characteristics of a community radio broadcasting service. We choose and evaluate eight popular broadcasting protocols on these characteristics, to find the protocols most suited for our application.

Proactive management of service instance pools for meeting service level agreements

Existing Grid schedulers focus on allocating resources to jobs as per the resource requirements expressed by end-users. This demands detailed knowledge of application behavior for different resource configurations on the part of end-users. Additionally, this model incurs significant delay in terms of the provisioning overhead for each request. In contrast, for interactive workloads, services are commonly pre-configured by an application server according to long-term steady-state requirements. In this paper, we propose a framework for bridging the gap between these two extremes. We target application services beyond simple interactive workloads, such as a parallel numeric application. In our approach, end users are shielded from lower-level resource configuration details and deal only with service metrics like average response time, expressed as SLAs. These SLAs are then translated into concrete resource allocation decisions. Since demand for a service fluctuates over time, static pre-configurations may not maximize utility of the common pool of resources. Our approach involves dynamic re-provisioning to achieve maximum utility, while accounting for overheads incurred during re-provisioning. We find that it is not always beneficial to re-provision resources according to perceived benefits and propose a model for calculating the optimal amount of re-provisioning for a particular scenario.

A LAYERED FRAMEWORK FOR CONNECTING CLIENT OBJECTIVES AND RESOURCE CAPABILITIES

In large-scale, distributed systems such as Grids, an agreement between a client and a service provider specifies service level objectives both as expressions of client requirements and as provider assurances. From an application perspective, these objectives should be expressed in a high-level, service or application-specific manner rather than requiring clients to detail the necessary resources. Resource providers on the other hand, expect low-level, resource-specific performance criteria that are uniform across applications and can be easily interpreted and provisioned. This paper presents a framework for service management that addresses this gap between high-level specification of client performance objectives and existing resource management infrastructures. The paper identifies three levels of abstraction for resource requirements a service provider needs to manage, namely: detailed specification of raw resources, virtualization of heterogeneous resources as abstract resources, and performance objectives at an application level. The paper also identifies three key functions for managing service-level agreements, namely: translation of resource requirements across abstraction layers, arbitration in allocating resources to client requests, and aggregation and allocation of resources from multiple lower-level resource managers. One or more of these key functions may be present at each abstraction layer of a service-level manager. Thus, layering and the composition of these functions across abstraction layers enables modeling of a wide array of management scenarios. The framework we present uses service metadata and/or service performance models to map client requirements to resource capabilities, uses business value associated with objectives to arbitrate between competing requests, and allocates resources based on previously negotiated agreements. We instantiate this framework for three different scenarios and explain how the architectural principles we introduce are used in the real-word.