Marvin H. Solomon

Matchmaking: distributed resource management for high throughput computing

Conventional resource management systems use a system model to describe resources and a centralized scheduler to control their allocation. We argue that this paradigm does not adapt well to distributed systems, particularly those built to support high throughput computing. Obstacles include heterogeneity of resources, which make uniform allocation algorithms difficult to formulate, and distributed ownership, leading to widely varying allocation policies. Faced with these problems, we developed and implemented the classified advertisement (classad) matchmaking framework, a flexible and general approach to resource management in distributed environment with decentralized ownership of resources. Novel aspects of the framework include a semi structured data model that combines schema, data, and query in a simple but powerful specification language, and a clean separation of the matching and claiming phases of resource allocation. The representation and protocols result in a robust, scalable and flexible framework that can evolve with changing resources. The framework was designed to solve real problems encountered in the deployment of Condor, a high throughput computing system developed at the University of Wisconsin-Madison. Condor is heavily used by scientists at numerous sites around the world. It derives much of its robustness and efficiency from the matchmaking architecture.

Shoring up persistent applications

SHORE (Scalable Heterogeneous Object REpository) is a persistent object system under development at the University of Wisconsin. SHORE represents a merger of object-oriented database and file system technologies. In this paper we give the goals and motivation for SHORE, and describe how SHORE provides features of both technologies. We also describe some novel aspects of the SHORE architecture, including a symmetric peer-to-peer server architecture, server customization through an extensible value-added server facility, and support for scalability on multiprocessor systems. An initial version of SHORE is already operational, and we expect a release of Version 1 in mid-1994.

Policy driven heterogeneous resource co-allocation with Gangmatching

Dynamic, heterogeneous and distributively owned resource environments present unique challenges to the problems of resource representation, allocation and management. Conventional resource management methods that rely on static models of resource allocation policy and behavior fail to address these challenges. We previously argued that Matchmaking provides an elegant and robust solution to resource management in such dynamic and federated environments. However, Matchmaking is limited by its purely bilateral formalism of matching a single customer with a single resource, precluding more advanced resource management services such as co-allocation. In this paper, we present Gangmatching, a multilateral extension to the Matchmaking model, and discuss the Gangmatching model and its associated implementation and performance issues in context of a real-world license management co-allocation problem.

Resource management through multilateral matchmaking

Federated distributed systems present new challenges to resource management, which cannot be met by conventional systems that employ relatively static resource models and centralized allocators. We previously argued that matchmaking provides an elegant and robust resource management solution for these highly dynamic environments (R. Raman et al., 1998). Although powerful and flexible, multiparty policies (e.g., co-allocation) cannot be accommodated by matchmaking. The authors present Gang-Matching, a multilateral matchmaking formalism to address this deficiency.

Matchmaking: An extensible framework for distributed resource management

Federated distributed systems present new challenges to resource management. Conventional resource managers are based on a relatively static resource model and a centralized allocator that assigns resources to customers. Distributed environments, particularly those built to support high-throughput computing (HTC), are often characterized by distributed management and distributed ownership. Distributed management introduces resource heterogeneity: Not only the set of available resources, but even the set of resource types is constantly changing. Distributed ownership introduces policy heterogeneity: Each resource may have its own idiosyncratic allocation policy. To address these problems, we designed and implemented the Matchmaking resource management framework. Customers and resources are all described by classified advertisements (classads) written in a simple but powerful formal language that describes their attributes and allocation policies. A Matchmaker server uses a policy-independent matching operation to discover pairings. It notifies the parties to the match, which use a separate, bilateral claiming protocol to confirm the allocation. The resulting framework is robust, scalable and flexible, and can evolve with changing resources. Matchmaking is the core of the Condor High Throughput Computing System developed at the University of Wisconsin — Madison. Condor is a production-quality system used by scientists and engineers at sites around the world. Condor derives much of its flexibility, robustness and efficiency from the matchmaking architecture. We describe the use of matchmaking in Condor, presenting several examples that illustrate its flexibility and expressiveness.

The Roscoe distributed operating system

Roscoe is an operating system implemented at the University of Wisconsin that allows a network of microcomputers to cooperate to provide a general-purpose computing facility. After presenting an overview of the structure of Roscoe, this paper reports on experience with Roscoe and presents several problems currently being investigated by the Roscoe project.

A Note on Enumerating Binary Trees

Abstract : Algorithms have been presented for computing a bijection between the set of binary trees on n nodes and an initial segment of the positive integers. A more complicated algorithm was also presented that computes a different bijection, claiming that their algorithm is more efficient and has advantages if a sequence of several consecutive trees is required. A modification of the first algorithm is presented that is simpler than the first and as efficient as the second. Also given is a new linear-time algorithm for transforming a tree into its successor in the natural ordering of binary trees.

The CSNET name server

Abstract CSNET is a project designed to facilitate electronic data communication among academic computer science departments and other groups doing computer-science research in the United States. CSNET will provide communications facilities for electronic mail and file transfer between users of computers connected to a variety of networks. For the system to be simple and easy to use, users must be able to identify each other to the system in a way that is natural to them and which does not require them to understand the details of network organization or to memorize cryptic names. To this end CSNET is implementing a name server service, composed of programs and data residing on a central Service Host computer and on individual member hosts of CSNET. This paper describes the architecture of the name server and discusses the considerations that lead to its design.

An approach to support automatic generation of user interfaces

In traditional interactive programming environments, each application individually manages its interaction with the human user. The result is duplication of effort in implementing user interface code and nonuniform—hence confusing—input conventions. This paper presents an approach to support automatic generation of user interfaces in environments based on algebraic languages. The approach supports the editing model of interaction, which allows a user to view all applications as data that can be edited. An application interacts with a user by submitting variables (of arbitrary types) to a dialogue manager, which displays their presentations to the user and offers type-directed editing of these presentations. Applications and dialogue managers communicate through a protocol that allows a presentation to be kept consistent with the variable it displays. A particular implementation of the approach, called Dost, has been constructed for the Xerox development environment and the Mesa programming language. Dost is used as a concrete example to describe the editing model, the primitives to support it, and our preliminary experience with these primitives. The approach is compared with related work, its shortcomings are discussed, and suggestions for future work are made.

Type definitions with parameters

It has long been known that recursively defined types in a highly typed language such as Algol 68 or Pascal may be tested for structural equivalence by the same algorithm that compares finite automata [5,11]. Several authors (for example, [3,8,9,16]) have proposed that classes of types be simultaneously defined by the use of parameterized type definitions, such asType list(x) = record val:x; next:↑list(x) end .This paper shows that unless the use of such parameterized definitions is restricted, new (unparameterized) types may be defined which more closely resemble deterministic context-free languages. In fact, the equivalence problem for such types becomes as hard as the (currently unsolved) deterministic pushdown automaton equivalence problem. Several restrictions on type definitions are considered which allow known equivalence algorithms to be applied.