Mason J. Katz

NPACI Rocks: tools and techniques for easily deploying manageable Linux clusters

High‐performance computing clusters (commodity hardware with low‐latency, high‐bandwidth interconnects) based on Linux are rapidly becoming the dominant computing platform for a wide range of scientific disciplines. Yet, straightforward software installation, maintenance, and health monitoring for large‐scale clusters has been a consistent and nagging problem for non‐cluster experts. The NPACI Rocks distribution takes a fresh perspective on management and installation of clusters to dramatically simplify software version tracking and cluster integration.

NPACI: rocks: tools and techniques for easily deploying manageable Linux clusters

High-performance computing clusters (commodity hardware with low-latency, high-bandwidth interconnects) based on Linux, are rapidly becoming the dominant computing platform for a wide range of scientific disciplines.Yet, straightforward software installation, maintenance, and health monitoring for large-scale clusters has been a consistent and nagging problem for non-cluster experts.The NPACI Rocks toolkit takes a fresh perspective on management and installation of clusters to dramatically simplify software version tracking, and cluster integration.The toolkit incorporates the latest Red Hat distribution (including security patches) with additional cluster-specific software.Using the identical software tools used to create the base distribution, users can customize and localize Rocks for their site.Strong adherence to widely-used (\emph{de facto}) tools allows Rocks to move with the rapid pace of Linux development.Version 2.1 of the toolkit is available for download and installation.To date, 10 clusters spread among 5 institutions have been built using this toolkit.

Rolls: modifying a standard system installer to support user-customizable cluster frontend appliances

The Rocks toolkit uses a graph-based framework to describe the configuration of all node types (termed appliances) that make up a complete cluster. With hundreds of deployed clusters, our turnkey systems approach has shown to be quite easily adapted to different hardware and logical node configurations. However, the Rocks architecture and implementation contains a significant asymmetry: the graph definition of all appliance types except the initial frontend can be modified and extended by the end-user before installation. However, frontends can be modified only afterward by hands-on system administration. To address this administrative discontinuity between nodes and frontends, we describe the design and implementation of Rolls. First and foremost, Rolls provide both the architecture and mechanisms that enable the end-user to incrementally and programmatically modify the graph description for all appliance types. New functionality can be added and any Rocks-supplied software component can be overwritten or removed simply by inserting the desired Roll CD(s) at installation time. This symmetric approach to cluster construction has allowed us to shrink the core of the Rocks implementation while increasing flexibility for the end-user. Rolls are optional, automatically configured, cluster-aware software systems. Current add-ons include: scheduling systems (SGE, PBS), grid support (based on NSF Middleware Initiative), database support (DB2), Condor, integrity checking (Tripwire) and the Intel compiler. Community-specific Rolls can be and are developed by groups outside of the Rocks core development group.

The PRAGMA Testbed - Building a Multi-Application International Grid

This practices and experience paper describes the coordination, design, implementation, availability, and performance of the Pacific Rim Applications and Grid Middleware Assembly (PRAGMA) Grid Testbed. Applications in high-energy physics, genome annotation, quantum computational chemistry, wildfire simulation, and protein sequence alignment have driven the middleware requirements, and the testbed provides a mechanism for international users to share software beyond the essential, de facto standard Globus core. In this paper, we describe how human factors, resource availability and performance issues have affected the middleware, applications and the testbed design. We also describe how middleware components in grid monitoring, grid accounting, grid Remote Procedure Calls, grid-aware file systems, and grid-based optimization have dealt with some of the major characteristics of our testbed. We also briefly describe a number of mechanisms that we have employed to make software more easily available to testbed administrators.

Configuring Large High-Performance Clusters at Lightspeed: A Case Study

Over a decade ago, the TOP500 list was started as a way to measure supercomputers by their sustained performance on a particular linear algebra benchmark. Once reserved for the exotic machines and extremely well-funded centers and laboratories, commodity clusters now make it possible for smaller groups to deploy and use high performance machines in their own laboratories. This paper describes a weekend activity where two existing 128-node commodity clusters were fused into a single 256-node cluster for the specific purpose of running the benchmark used to rank the machines in the TOP500 supercomputer list. The resulting metacluster sits on the November 2002 list at position 233. A key differentiator for this cluster is that it was assembled, in terms of its software, from the NPACI Rocks open-source cluster toolkit as downloaded from the public website. The toolkit allows non-cluster experts to deploy and run supercomputer-class machines in a matter of hours instead of weeks or months. With the exception of recompiling the University of Tennessee’s Automatically Tuned Linear Algebra Subroutines (ATLAS) library with a recommended version of the GNU C compiler, this metacluster ran a “stock” Rocks distribution. Successful first-time deployment of the fused cluster was completed in a scant 6 h. Partitioning of the metacluster and restoration of the two 128-node clusters to their original configuration was completed in just over 40 min. This paper describes early (pre-weekend) benchmark activities to empirically determine reasonably good parameters for the High Performance Linpack (HPL) code on both Ethernet and Myrinet interconnects. It fully describes the physical layout of the machine, the description-based installation methods used in Rocks to re-deploy two independent clusters as a single cluster, and gives the benchmark results that were gathered over the 40-h period allotted for the complete experiment. In addition, we describe some of the on-line monitoring and measurement techniques that were employed during the experiment. Finally, we point out the issues uncovered with a commodity cluster of this size. The techniques presented in this paper truly bring supercomputers into the hands of the masses of computational scientists.

Lessons learned through driving science applications in the PRAGMA grid

This paper describes the coordination, design and implementation of the PRAGMA Grid. Applications in genomics, quantum mechanics, climate simulation, organic chemistry and molecular simulation have driven the middleware requirements, and the PRAGMA Grid provides a mechanism for science and technology teams to collaborate, for grids to interoperate and for international users to share software beyond the essential, de facto standard Globus core. Several middleware tools developed by researchers within PRAGMA have been deployed in the PRAGMA grid and this has enabled significant insights, improvements and new collaborations to flourish. In this paper, we describe how human factors, resource availability and performance issues have affected the middleware, applications and the grid design. We also describe how middleware components in grid monitoring, grid accounting and grid file systems have dealt with some of the major characteristics of our grid. We also briefly describe a number of mechanisms that we have employed to make software easily available to PRAGMA and global grid communities.

Beyond Beowulf Clusters

In the early ’90s, the Berkeley NOW Project under David Culler posited that groups of less capable machines could be used to solve scientific and other computing problems at a fraction of the cost of larger computers. In 1994, Donald Becker and Thomas Sterling worked to drive the costs even lower by adopting the then-fledgling Linux operating system to build Beowulf clusters at NASA’s Goddard Space Flight Center. By tying desktop machines together with open source tools such as PVM, MPI, and PBS, early clusters—which were often PC towers stacked on metal shelves with a nest of wires interconnecting them—fundamentally altered the balance of scientific computing. Before these first clusters appeared, distributed/parallel computing was prevalent at only a few computing centers, national laboratories, and a very few university departments. Since the introduction of clusters, distributed computing is now, literally, everywhere.

411 on scalable password service

In this paper we present 411, a password distribution system for high performance environments that provides security and scalability. We show that existing solutions such as NIS and Kerberos do not provide sufficient performance in large, tightly coupled systems such as computational clusters. Unlike existing single-signon services, the 411 design removes the need for communication during password lookup by using aggressive replication techniques. We demonstrate the use of shared keys to efficiently protect user information, and the careful management of system wide consistency and fault tolerance. A theoretical analysis of the behavior of 411 is matched with quantitative evidence of its performance and suitability to a clustered environment. We further show the system effectively responds to stress by simulating 50% message loss on a 60-node cluster. This protocol is currently used worldwide in hundreds of Rocks-based production systems to provide password and login information service.

Incorporation of Middleware and Grid Technologies to Enhance Usability in Computational Chemistry Applications

High performance computing, storage, visualization, and database infrastructures are increasing in complexity as research moves towards grid-based computing, often pushing breakthrough computational capabilities beyond the reach of scientists due to the time needed to harness the infrastructure. Hiding the underlying complexity of networked resources becomes essential if scientists are to utilize these resources in a time-effective manner. There are a myriad of solutions that have been proposed, ranging from underlying grid glue, to fully integrated problem solving environments. In this work, we discuss a workflow management system that is fully integrated with emerging grid standards but can be dynamically reconfigured. Through defined XML schema to describe both resources and application codes and interfaces, careful implementation with emerging grid standards and user-friendly interfaces, a “pluggable” event-driven model is created where grid-enabled services can be composed to form more elaborate pipelines of information processing, simulation, and visual analysis.