Paul Z. Kolano

Surfer: an extensible pull-based framework for resource selection and ranking

Grid computing aims to connect large numbers of geographically and organizationally distributed resources to increase computational power resource utilization, and resource accessibility. In order to effectively utilize Grids, users need to be connected to the best available resources at any given time. As Grids are in constant flux, users cannot be expected to keep up with the configuration and status of the Grid, thus they must be provided with automatic resource brokering for selecting and ranking resources meeting constraints and preferences they specify. This work presents a new OGSI-compliant resource selection and ranking framework called Surfer that has been implemented as part of NASAs Information Power Grid (IPG) project. Surfer is highly extensible and may be integrated into any Grid environment by adding information providers knowledgeable about that environment. Surfer invisibly and seamlessly correlates results from different providers into a single unified view seen by the user.

A unified framework for periodic, on-demand, and user-specified software information

Although grid computing can increase the number of resources available to a user, not all resources on the grid may have a software environment suitable for running a given application. To provide users with the necessary assistance for selecting resources with compatible software environments and/or for automatically establishing such environments, an accurate source of information about the software installed across the grid is needed. Existing software information services and general-purpose information frameworks are inadequate for this task. This paper presents a new OGSI-compliant software information service that has been implemented as part of NASAs Information Power Grid project. This service is built on top of a general framework for reconciling information from periodic, on-demand, and user-specified sources. Information is retrieved using standard XPath queries over a single unified namespace independent of the informations source. Two consumers of the provided software information, the IPG Resource Broker and the IPG Naturalization Service, are briefly described.

Mesh: secure, lightweight grid middleware using existing SSH infrastructure

Grid computing promises gains in effective computational power, resource utilization, and resource accessibility, but in order to achieve these gains, organizations must deploy grid middleware that, in most cases, does not adhere to fundamental security principles. This paper introduces a new lightweight grid middleware called Mesh, which is based on the addition of a single sign-on capability to the built-in public key authentication mechanism of SSH using system call interposition. The initial Mesh implementation is compatible with approximately 90% of the worlds SSH servers and any SSH client that supports public key authentication. Resources maybe added to a Mesh-based grid in a matter of minutes using just five small files and two environment variable settings. Mesh adheres to fundamental security principles and was designed to be compatible with strong security mechanisms including two-factor authentication, SSH bastions, and restrictive firewalls. Mesh uses a remote command model, which is based on the syntax and commands already understood by users, thus requires no additional knowledge to utilize effectively. Several existing services have been integrated with Mesh to provide resource discovery and query, high performance file transfer, and job management.

A scalable aural-visual environment for security event monitoring, analysis, and response

Intrusion detection systems gather large quantities of host and network information in an attempt to detect and respond to attacks against an organization. The widely varying nature of attacks makes humans essential for analysis, but the sheer volume of data can quickly overwhelm even experienced analysts. Existing approaches utilize visualization to provide rapidly comprehensible representations of the data, but fail to scale to real-world environments due to unrealistic data handling and lack of response facilities. This paper introduces a new tool for security event monitoring, analysis, and response called Savors. Savors provides suitable scalability by utilizing three additional areas of computing. High-end computing brings large amounts of on-demand processing to bear on the problem. Auralization allows both monitoring and analysis to be performed in parallel. Finally, grid computing provides the basis for remote data access and response capabilities with seamless and secure access to organization resources.

Facilitating the Portability of User Applications in Grid Environments

Grid computing promises the ability to connect geographically and organizationally distributed resources to increase effective computational power, resource utilization, and resource accessibility. For grid computing to be successful, users must be able to easily execute the same application on different resources. Different resources, however, may be administered by different organizations with different software installed, different file system structures, and different default environment settings. Even within the same organization, the set of software installed on a given resource is in constant flux with additions, upgrades, and removals. Users cannot be expected to understand all of the idiosyncrasies of each resource they may wish to execute jobs on, thus must be provided with automated assistance. This paper describes a new OGSI-compliant grid service (the Naturalization Service) that has been implemented as part of NASA’s Information Power Grid (IPG) project to automatically establish the execution environment for user applications.

High performance reliable file transfers using automatic many-to-many parallelization

Shift is a lightweight framework for high performance local and remote file transfers that provides resiliency across a wide variety of failure scenarios. Shift supports multiple file transport protocols with automatic selection of the most appropriate mechanism between each pair of participating hosts allowing it to adapt to heterogeneous clients with differing software and network access restrictions. File system information is gathered from clients and servers to detect file system equivalence and enable path rewriting so that multiple clients can be automatically spawned in parallel to carry out both single and multi-file transfers to multiple servers selected according to load and availability. This improves both reliability and performance by eliminating single points of failure and overcoming single system bottlenecks. End-to-end integrity is provided using cryptographic hashes at the source and destination with support for partial file retransmission of only corrupted portions. This paper presents the design and implementation of Shift and details the mechanisms utilized to enhance the reliability and performance of file transfers.

Transparent Optimization of Parallel File System I/O via Standard System Tool Enhancement

Standard system tools employed by users on a daily basis do not take full advantage of parallel file system I/O bandwidth and do not understand associated idiosyncrasies such as Lustre striping. This can lead ton on-optimal utilization of both the users time and system resources. This paper describes a set of modifications made to existing tools that increase parallelism and automatically handle striping. These modifications result insignificant performance gains in a transparent manner with maximum speedups of 27×, 15×, and 31× for parallelized cp, tar creation, and tar extraction, respectively.

Maintaining high performance communication under least privilege using dynamic perimeter control

From a security standpoint, it is preferable to implement least privilege network security policies in which only the bare minimum of TCP/UDP ports on internal hosts are accessible from outside the perimeter. Unfortunately, organizations with such policies can no longer communicate using common multiport protocols that require randomly chosen ports for auxiliary connections. This paper introduces a new approach for maintaining such communication under least privilege while achieving maximum performance. By dynamically modifying perimeter ACLs, inbound auxiliary connections are only allowed through the perimeter at exactly the times required. These modifications are made transparently to external users and with minimal changes to internal configuration. A prototype implementation of the Dynamic Perimeter Enforcement system, called Diaper, has been implemented and tested with several applications.

Automatically encapsulating HPC best practices into data transfers

This paper presents the Shift automated transfer tool and the mechanisms it employs to achieve better performance while preserving the stability of HPC environments. Shift encapsulates best practices understood by domain experts during transfers so that scientists can focus on their science without the need to study file transports, resource management, and file systems as well. Shift understands how to utilize the variety of transports that might be deployed throughout a widely distributed user base, how to maximize the performance achievable by each, and the scenarios in which each is most effective. Shift understands which resources are available in a particular HPC environment and how to utilize them for significant performance increases while preventing resource exhaustion. Finally, Shift understands the file systems to which and from which files may be transferred and the nuances to their use that affect performance and stability behind the scenes.

Dynamic load balancing of SSH sessions using user-specific selection policies

Ballast is a tool for balancing user load across SSH servers based on various criteria such as CPU load and system availability. It includes a load balancing client, a lightweight data server, and a data collection agent. Ballast is invoked as part of the SSH login process, so has access to the user name while making balancing decisions, which is not available in traditional load balancing approaches. This gives Ballast the unique ability to perform user-specific load balancing. This paper presents the Ballast architecture and examines the benefits of involving user-specific criteria in the balancing process. Two approaches for utilizing user information based on prediction and dynamic load metrics are analyzed using trace-based simulation and are found to have significant benefits when combined.