Michael Wan

Data grids, collections, and grid bricks

Data grids federate storage resources. They provide a logical name space that can be used to register digital entities, a storage repository abstraction for manipulating data, and a high level abstraction for supporting user-selected interfaces. Data grids can be used to build persistent collections. Data can be stored across multiple types of storage systems with persistent copies kept in archives. Persistent identifiers can be kept in the logical name space. By integrating grid bricks (commodity based disk caches) with archival storage systems, one can assemble a data management environment that supports both interactive access (data picking) and long-term persistent storage. Examples of the creation of interactive data picking environments will be given that integrate grid brick technology with large-scale archives.

Data Grids, Digital Libraries, and Persistent Archives: An Integrated Approach to Sharing, Publishing, and Archiving Data

The integration of grid, data grid, digital library, and preservation technology has resulted in software infrastructure that is uniquely suited to the generation and management of data. Grids provide support for the organization, management, and application of processes. Data grids manage the resulting digital entities. Digital libraries provide support for the management of information associated with the digital entities. Persistent archives provide long-term preservation. We examine the synergies between these data management systems and the future evolution that is required for the generation and management of information.

A simple mass storage system for the SRB data grid

The functionality that is provided by Mass Storage Systems can be implemented using data grid technology. Data grids already provide many of the required features, including a logical name space and a storage repository abstraction. We demonstrate how management of tape resources can be integrated into data grids. The resulting infrastructure has the ability to manage archival storage of digital entities on tape or other media, while maintaining copies on distributed, remote disk caches that can be accessed through advanced discovery mechanisms. Data grids provide additional levels of data management including the ability to aggregate data into containers before storage on tape, and the ability to migrate collections across a hierarchy of storage devices.

A batch scheduler for the Intel Paragon with a non-contiguous node allocation algorithm

As the system usage model for scalable parallel processors evolves from the singleuser, dedicated access model to a multi-user production environment, a versatile batch scheduler is needed to match user requirements to system resources. This paper describes the design and performance of a batch scheduler for the Intel Paragon system that addresses the issues associated with a multi-user production environment, including scheduling for heterogeneous nodes, scheduling for long-running jobs, scheduling for large jobs, prime/non-prime time modes, and node allocation schemes. The Modified 2-D Buddy system (M2DB) for non-contiguous node allocation is introduced and studied in this paper.

Production Storage Resource Broker Data Grids

International data grids are now being built that support joint management of shared collections. An emerging strategy is to build multiple independent data grids, each managed by the local institution. The data grids are then federated to enable controlled sharing of files. We examine the management issues associated with maintaining federations of production data grids, including management of access controls, coordinated sharing of name spaces, replication of data between data grids, and expansion of the data grid federation.

Universal view and open policy: Paradigms for collaboration in data grids

Large-scale Data Grid Systems (LDGS) facilitate collaborative sharing of large collections (Petabytes and100s of millions of objects) containing files, databases and data streams that are geographically distributed across heterogeneous resources and multiple administrative domains. LDGS provide a “universal view” of the distributed data, resources, users and methods and hide the idiosyncrasies and the heterogeneity of the underlying infrastructure and protocols - enhancing user collaborations. To improve transparency, an “open policy” system is needed by which data providers and administrators can describe the exact processes and policies that implement LDGS services. We consider policies and processes as the essential defining characteristics of a productive LDGS collaboration. We have implemented an LDGS, called integrated Rule-Oriented Data Systems (iRODS), which provides a universal view while enabling an open policy environment for publishing descriptions of the available services. The open policy environment is supported by a distributed workflow/rule engine. The services are encoded as rules in a high-level workflow language that transparently describes the underlying functionality. Well-defined semantics are used to control the composition of the workflow functions, called micro-services, to map to the desired client-level actions. In this paper, we describe the iRODS system from the “universal view” and “open policy” perspective and show its scalability for managing more than 10 million files.