Selim Kalayci

Enabling Interoperability among Meta-Schedulers

Grid computing supports shared access to computing resources from cooperating organizations or institutes in the form of virtual organizations. Resource brokering middleware, commonly known as a meta-scheduler or a resource broker, matches jobs to distributed resources. Recent advances in meta- scheduling capabilities are extended to enable resource matching across multiple virtual organizations. Several architectures have been proposed for interoperating meta-scheduling systems. This paper presents a hybrid approach, combining hierarchical and peer-to-peer architectures for flexibility and extensibility of these systems. A set of protocols are introduced to allow different meta-scheduler instances to communicate over Web Services. Interoperability between three heterogeneous and distributed organizations (namely, BSC, FIU, and IBM), each using different meta-scheduling technologies, is demonstrated under these protocols and resource models.

Transparent grid enablement of weather research and forecasting

The impact of hurricanes is so devastating throughout different levels of society that there is a pressing need to provide a range of users with accurate and timely information that can enable effective planning for and response to potential hurricane landfalls. The Weather Research and Forecasting (WRF) code is the latest numerical model that has been adopted by meteorological services worldwide. The current version of WRF has not been designed to scale out of a single organizations local computing resources. However, the high resource requirements of WRF for fine-resolution and ensemble forecasting demand a large number of computing nodes, which typically cannot be found within one organization. Therefore, there is a pressing need for the Grid-enablement of the WRF code such that it can utilize resources available in partner organizations. In this paper, we present our research on Grid enablement of WRF by leveraging our work in transparent shaping, GRID superscalar, profiling, code inspection, code modeling, meta-scheduling, and job flow management.

Design and Implementation of a Fault Tolerant Job Flow Manager Using Job Flow Patterns and Recovery Policies

Currently, many grid applications are developed as job flows that are composed of multiple jobs. The execution of job flows requires the support of a job flow manager and a job scheduler. Due to the long running nature of job flows, the support for fault tolerance and recovery policies is especially important. This support is inherently complicated due to the sequencing and dependency of jobs within a flow, and the required coordination between workflow engines and job schedulers. In this paper, we describe the design and implementation of a job flow manager that supports fault tolerance. First, we identify and label job flow patterns within a job flow during deployment time. Next, at runtime, we introduce a proxy that intercepts and resolves faults using job flow patterns and their corresponding fault-recovery policies. Our design has the advantages of separation of the job flow and fault handling logic, requiring no manipulation at the modeling time, and providing flexibility with respect to fault resolution at runtime. We validate our design with a prototypical implementation based on the ActiveBPEL workflow engine and GridWay Meta-scheduler, and Montage application as the case study.

Enabling Autonomic Meta-Scheduling in Grid Environments

Grid computing supports workload execution on computing resources that are shared across a set of collaborative organizations. At the core of workload management for grid computing is a software component, called meta-scheduler or grid resource broker, that provides a virtual layer on top of heterogeneous grid middleware, schedulers, and resources. Meta-schedulers typically enable end-users and applications to compete over distributed shared resources through the use of one or more instances of the same meta-scheduler, in a centralized or distributed manner, respectively. We propose an approach to enabling autonomic meta-scheduling through the use of a new communication protocol that -if adopted by different meta-schedulers or by the applications using them- can improve the workload execution while avoiding potential chaos, which can be resulted from blind competition over resources. This can be made possible by allowing the meta- schedulers and/or their applications to engage in a process to negotiate their roles (e.g., consumer, provider, or both), scheduling policies, service-level agreement, etc. To show the feasibility of our approach, we developed a prototype that enables some preliminary autonomic management among three different meta-schedulers, namely, GridWay, eNANOS, andTDWB.

Managing Faults for Distributed Workflows over Grids

Grid applications composed of multiple, distributed jobs are common areas for applying Web-scale workflows. Workflows over grid infrastructures are inherently complicated due to the need to both functionally assure the entire process and coordinate the underlying tasks. Often, these applications are long-running, and fault tolerance becomes a significant concern. Transparency is a vital aspect to understanding fault tolerance in these environments.

A Self-Configuring Communication Virtual Machine

Todays communication-based applications are mostly crafted in a stovepipe development paradigm, which is inflexible to be used by various domain-specific applications and costly in the development phase. In a previous paper, we proposed a new design called CVM (communication virtual machine) to overcome these problems by having a high-level API which can be reused and extended easily for user-centric applications in any domain. Within CVM framework, we came across a practical issue, which is actually the case for any end- to-end multimedia communication, namely the NAT-traversal (network address translation) problem that limits the reliability and availability of CVM and variants of CVM. In this paper, we explain about the necessity of self-configuration for the NAT-traversal problem in end-to-end communications, and propose a solution within the core CVM framework.

Runtime Fault-Handling for Job-Flow Management in Grid Environments

The execution of job flow applications is a reality today in academic and industrial domains. In this paper, we propose an approach to adding self-healing behavior to the execution of job flows without the need to modify the job flow engines or redevelop the job flows themselves. We show the feasibility of our non-intrusive approach to self-healing by inserting a generic proxy to an existing two-level job-flow management system, which employs job flow based service orchestration at the upper level, and service choreography at the lower level. The generic proxy is inserted transparently between these two layers so that it can intercept all their interactions. We developed a prototype of our approach in a real Grid environment to show how the proxy facilitates runtime handling for failure recovery.

Optimization Patterns for the Decentralized Orchestration of Parameter-Sweep Workflows

A large and diverse group of computational scientific research efforts deal with parameterized studies, in which same or similar computational tools are applied on different sets of data. Such uniform and well-defined analysis efforts can be encapsulated as parameter-sweep workflows. Due to computation and data intensive nature, resources that span across multiple domains may be needed for timely and efficient execution of this type of workflows. In our previous studies, we have designed and developed techniques to orchestrate the execution of large-scale workflows in a decentralized and adaptive manner. Through the usage of generic workflow patterns, centralized orchestration of workflows are transformed into decentralized and adaptive orchestration without modifying the business logic of the workflow. In this study, we propose some additional optimization patterns specific to characteristics and requirements of parameter-sweep workflows. By exploiting the general characteristics of parameter-sweep workflows, we provide ways to reduce control and data overheads associated with the decentralized orchestration. We also discuss some implementation issues that arise from the adoption of these optimization patterns.