Anindya Basu

U-Net: a user-level network interface for parallel and distributed computing

The U-Net communication architecture provides processes with a virtual view of a network interface to enable userlevel access to high-speed communication devices. The architecture, implemented on standard workstations using offthe-shelf ATM communication hardware, removes the kernel from the communication path, while still providing full protection. The model presented by U-Net allows for the construction of protocols at user level whose performance is only limited by the capabilities of network. The architecture is extremely flexible in the sense that traditional protocols like TCP and UDP, as well as novel abstractions like Active Messages can be implemented efficiently. A U-Net prototype on an 8node ATM cluster of standard workstations offers 65 microseconds round-trip latency and 15 Mbytes/sec bandwidth. It achieves TCP performance at maximum network bandwidth and demonstrates performance equivalent to Meiko CS-2 and TMC CM-5 supercomputers on a set of Split-C benchmarks.

Low-latency communication over ATM networks using active messages

Todays communication architectures for parallel machines reduce communication overheads and latencies by over an order of magnitude. However, carrying over these techniques to workstation clusters connected by an ATM network presents major design challenges. We discuss the differences in communication characteristics between workstation clusters built from standard hardware and software components and state-of-the-art multiprocessors, and then evaluate a prototype implementation of an active message communication layer. Application round-trip latencies of about 50 microseconds for small messages roughly compare to a similar implementation on the Thinking Machines CM-5 multiprocessor. >

Simulating Reliable Links with Unreliable Links in the Presence of Process Crashes

We study the effect of link failures on the solvability of problems in distributed systems. In particular, we address the following question: given a problem that can be solved in a system where the only possible failures are process crashes, is the problem still solvable if links can also fail by losing messages? The answer depends on several factors, including the synchrony of the system, the model of link failures, the maximum number of process failures, and the nature of the problem to be solved. In this paper, we focus on asynchronous systems (results concerning synchronous systems will be described in a companion paper). The set of problems solvable in asynchronous systems with process crashes include Reliable, FIFO, and Causal Broadcast, and their uniform counterparts [Bir85, HT94], as well as Approximate Agreement [DLP+86], Renaming [ABND+90], and k-set Agreement [Cha90]. The question is whether such problems remain solvable (and if so, how) if we add link failures. We consider two models of lossy links: eventually reliable and fair lossy. Roughly speaking, they have the following properties: with an eventually reliable link, there is a time after which all messages sent are eventually received (messages sent before that time may be lost). Such a link can lose only a finite (but unbounded) number of messages. With a fair lossy link, if an infinite number of messages are sent, an infinite subset of these messages is received. Such a link can lose an infinite number of messages. Clearly, any algorithm that works with fair lossy links also works with eventually reliable links. Thus, to make our results as strong as possible, we assume eventually reliable links when we prove impossibility results, and fair lossy links when we show problems to be solvable. 3 Since an eventually reliable link can lose only a finite number of messages, it may appear that one can mask these message losses by repeatedly sending copies of each message, or by piggybacking on each message all the messages that were previously sent. Such a scheme is highly inefficient, but it does seem to simulate a reliable link. So it appears that, in principle, any problem that can be solved in a system with process crashes and reliable links, remains solvable in a system with process crashes and eventually reliable links.

ATM and fast Ethernet network interfaces for user-level communication

Fast Ethernet and ATM are two attractive network technologies for interconnecting workstation clusters for parallel and distributed computing. This paper compares network interfaces with and without programmable co-processors for the two types of networks using the U-Net communication architecture to provide low-latency and high-bandwidth communication. U-Net provides protected, user-level access to the network interface and offers application-level round-trip latencies as low as 60 /spl mu/sec over Fast Ethernet and 90 /spl mu/sec over ATM. The design of the network interface and the underlying network fabric have a large bearing on the U-Net design and performance. Network interfaces with programmable co-processors can transfer data directly to and from user space while others require aid from the operating system kernel. The paper provides detailed performance analysis of U-Net for Fast Ethernet and ATM, including application-level performance on a set of Split-C parallel benchmarks. These results show that high-performance computing is possible on a network of PCs connected via Fast Ethernet.

Low-Latency Communication over Fast Ethernet

Fast Ethernet (100Base-TX) can provide a low-cost alternative to more esoteric network technologies for high-performance cluster computing. We use a network architecture based on the U-Net approach to implement low-latency and high-bandwidth communication over Fast Ethernet, with performance rivaling (and in some cases exceeding) that of 155 Mbps ATM. U-Net provides protected, user-level access to the network interface and enables application-level round-trip latencies of less than 60μs over Fast Ethernet.

Low-latency communication over ATM Networks using Active Messages

Recent developments in communication architectures for parallel machines have made significant progress and reduced the communication overheads and latencies by over an order of magnitude as compared to earlier proposals. This paper examines whether these techniques can carry over to clusters of workstations connected by an ATM network even though clusters use standard operating system software, are equipped with network interfaces optimized for stream communication, do not allow direct protected user-level access to the network, and use networks without reliable transmission or flow control. In a first part, this paper describes the differences in communication characteristics between clusters of work stations built from standard hardware and software components and state-of-the-art multiprocessors. The lack of flow control and of operating system coordination affects the communication layer design significantly and requires larger buffers at each end than on multiprocessors. A second part evaluates a prototype implementa tion of the low-latency Active Messages communication. model an a Sun workstation cluster interconnected by an A3`M network. Measurements show application-to-application latencies of about 20 microseconds for small messages which is roughly comparable to the Active Messages implementation on the Thinking Machines CM-5 multiprocessor.

Crash failures vs. crash + link failures

We study the effect of link failures on the solvability of problems in distributed systems. In particular, we address the following question: given a problem that can be solved in a system where the only possible failures are process crashes, is the problem still solvable if links can also fail by losing messages? The answer depends on several factors, includlng the synchrony of the system, the model of link failures, the maximum number of process failures, and the nature of the problem to be solved.