Viraj Bhat

AutoMate: enabling autonomic applications on the grid

The increasing complexity, heterogeneity and dynamism of networks, systems, services applications have made our computational/information infrastructure brittle, unmanageable and insecure. This has necessitated the investigation of a new paradigm for design, development and deployment based on strategies used by biological systems to deal with complexity, heterogeneity, and uncertainty, i.e. autonomic computing. This paper introduces the AutoMate project and describes its key components. The overall objective of AutoMate is to investigate key technologies to enable the development of autonomic grid applications that are context aware and are capable of self-configuring, self-composing, self-optimizing and self-adapting. Specifically, it will investigate the definition of autonomic components, the development of autonomic applications as dynamic composition of autonomic components, and the design of key enhancements to existing grid middleware and runtime services to support these applications.

Enabling Self-Managing Applications using Model-based Online Control Strategies

The increasing heterogeneity, dynamism and uncertainty of emerging DCE (Distributed Computing Environment) systems imply that an application must be able to detect and adapt to changes in its state, its requirements and the state of the system to meet its desired QoS constraints. As system and application scales increase, ad hoc heuristic-based approaches to application adaptation and self-management quickly become insufficient. This paper builds on the Accord programming system for rule-based self-management and extends it with model-based control and optimization strategies. This paper also presents the development of a self-managing data streaming service based on online control using Accord. This service is part of a Grid-based fusion simulation workflow consisting of long-running simulations, executing on remote supercomputing sites and generating several terabytes of data, which must then be streamed over a wide-area network for live analysis and visualization. The self-managing data streaming service minimize data streaming overheads on the simulations, adapt to dynamic network bandwidth and prevent data loss. An evaluation of the service demonstrating its feasibility is presented.

High performance threaded data streaming for large scale simulations

We have developed a threaded parallel data streaming approach using logistical networking (LN) to transfer multiterabyte simulation data from computers at NERSC to our local analysis/visualization cluster, as the simulation executes, with negligible overhead. Data transfer experiments show that this concurrent data transfer approach is more favorable compared with writing to local disk and later transferring this data to be post-processed. Our algorithms are network aware, and can stream data at up to 97 Mbs on a 100 Mbs link from CA to NJ during a live simulation, using less than 5% CPU overhead at NERSC. This method is the first step in setting up a pipeline for simulation workflow and data management.

An autonomic service architecture for self-managing grid applications

The scale, heterogeneity and dynamism of grid applications and environments require grid applications to be self-managing or autonomic. This paper presents the Accord autonomic services architecture that addresses this requirement. Accord enables service and application behaviors and their interactions to be dynamically specified and adapted using high-level rules, based on current application requirements, state and execution context. The design, implementation and evaluation of Accord are presented. An autonomic data streaming application is used to illustrate the self-managing behaviors enabled by Accord.

Autonomic oil reservoir optimization on the Grid

The emerging Grid infrastructure and its support for seamless and secure interactions is enabling a new generation of autonomic applications where the application components, Grid services, resources, and data interact as peers to manage, adapt and optimize themselves and the overall application. In this paper we describe the design, development and operation of a prototype of such an application that uses peer‐to‐peer interactions between distributed services and data on the Grid to enable the autonomic optimization of an oil reservoir. Copyright

A Self-Managing Wide-Area Data Streaming Service using Model-based Online Control

Efficient and robust data streaming services are a critical requirement of emerging Grid applications, which are based on seamless interactions and coupling between geographically distributed application components. Furthermore the dynamism of Grid environments and applications requires that these services be able to continually manage and optimize their operation based on system state and application requirements. This paper presents a design and implementation of such a self-managing data-streaming service based on online control strategies. A Grid-based fusion workflow scenario is used to evaluate the service and demonstrate its feasibility and performance.

Data management on the fusion computational pipeline

Fusion energy science, like other science areas in DOE, is becoming increasingly data intensive and network distributed. We discuss data management techniques that are essential for scientists making discoveries from their simulations and experiments, with special focus on the techniques and support that Fusion Simulation Project (FSP) scientists may need. However, the discussion applies to a broader audience since most of the fusion SciDACs, and FSP proposals include a strong data management component. Simulations on ultra scale computing platforms imply an ability to efficiently integrate and network heterogeneous components (computational, storage, networks, codes, etc.), and to move large amounts of data over large distances. We discuss the workflow categories needed to support such research as well as the automation and other aspects that can allow an FSP scientist to focus on the science and spend less time tending information technology.

A Middleware Substrate for Integrating Services on the Grid

In this paper we present the design, implementation and evaluation of the Grid-enabled Discover middleware substrate. The middleware substrate enables Grid infrastructure services provided by the Globus Toolkit (security, information, resource management, storage) to interoperate with collaboratory services provided by Discover (collaborative application access, monitoring, and steering). Furthermore, it enables users to seamlessly access and integrate local and remote services to synthesize customized middleware configurations on demand.

Autonomic Data Streaming for High-Performance Scientific Applications

CONTENTS 20.