Mark W. Goudreau

BSPlib: The BSP programming library

BSPlib is a small communications library for bulk synchronous parallel (BSP) programming which consists of only 20 basic operations. This paper presents the full definition of BSPlib in C, motivates the design of its basic operations, and gives examples of their use. The library enables programming in two distinct styles: direct remote memory access (DRMA) using put or get operations, and bulk synchronous message passing (BSMP). Currently, implementations of BSPlib exist for a variety of modern architectures, including massively parallel computers with distributed memory, shared memory multiprocessors, and networks of workstations. BSPlib has been used in several scientific and industrial applications; this paper briefly describes applications in benchmarking, Fast Fourier Transforms (FFTs), sorting, and molecular dynamics.

Towards efficiency and portability: programming with the BSP model

The Bulk-Synchronous Parallel (BSP) model was proposed by Valiant as a model for general-purpose parallel computation. The objective of the model is to allow the design of parallel programs that can be executed efficiently on a variety of architectures. While many theoretical arguments in support of the BSP model have been presented, the degree to which the model can be efficiently utilized on existing parallel machines remains unclear. To explore this question, we implemented s small library of BSP functions, called the Green BSP library, on several parallel platfotms. We also created a number of parallel applications based on this library. Here, we report on the performance of six of these applications on three different parallel platforms. Our preliminary results suggest that the BSP model can be used to develop efficient and portable programs for a range of machines and applications.

Efficient communication using total-exchange

A central question in parallel computing is to determine the extent to which one can write parallel programs using a high-level, general-purpose, and architecture-independent programming language and have them executed on a variety of parallel and distributed architectures without sacrificing efficiency. A large body of research suggests that, at least in theory, general-purpose parallel computing is indeed possible provided certain conditions are met: an excess of logical parallelism in the program, and the ability of the target architecture to efficiently realize balanced communication patterns. The canonical example of a balanced communication pattern is an h-relation, in which each processor is the origin and destination of at most h messages. A plethora of protocols has been designed for routing h-relations in a variety of networks. The goal has been to minimize the value of h while guaranteeing delivery of the messages within a time constant factor from optimal. In this paper we describe protocols that meet the most stringent efficiency requirement, namely delivery of messages within time that is a lower order additive term from the best achievable. Such protocols are called 1-optimal. While these protocols achieve 1-optimality only for heavily loaded networks, that is, for large values of h, they are remarkable for their simplicity in that they only use the total-exchange communication primitive. The total-exchange can be realized in many networks using very simple, contention-free, and extremely efficient schemes. The technical contribution of this paper is a protocol to route random h-relations in an N-processor network using /sup h///sub N/(1+o(1))+O(log log N) total-exchange rounds with high probability. Using message duplication, we can improve the bound to /sup h///sub N/(1+o(1))+O(log*N). This improves upon the /sup h///sub N/(1+o(1))+O(log N) bound of Gerbessiotis and Valiant. While our theoretical improvements are modest, our experimental results show an improvement over the protocol of A. Gerebessiotis and L.G. Valiant.<<ETX>>

Using recurrent neural networks to learn the structure of interconnection networks

A modified Recurrent Neural Network (RNN) is used to learn a Self-Routing Interconnection Network (SRIN) from a set of routing examples. The RNN is modified so that it has several distinct initial states. This is equivalent to a single RNN learning multiple different synchronous sequential machines. We define such a sequential machine structure as augmented and show that a SRIN is essentially an Augmented Synchronous Sequential Machine (ASSM). As an example, we learn a small six-switch SRIN. After training we extract the networks internal representation of the ASSM and corresponding SRIN.

Single-Message vs. Batch Communication

The selection of appropriate communication mechanisms is a key issue in parallel computing. We argue that the current emphasis on single-message communication has led to inefficient systems and unnecessarily confusing code. In contrast, batch communication has substantial implementation advantages, is suitable for almost all parallel applications, and encourages a programming paradigm that is easy to reason about.

ROUTING IN RANDOM MULTISTAGE INTERCONNECTIONS NETWORKS: COMPARING EXHAUSTIVE SEARCH, GREEDY AND NEURAL NETWORK APPROACHES

The problem of establishing point-to-point communication routes in a random multistage interconnection network (RMIN) is addressed. A neural network routing scheme is presented. This routing scheme is compared to two more traditional routing techniques—namely, exhaustive search routing and greedy routing. The main criterion that is examined is the ability of each routing methodology to solve routing problems. Results are obtained through simulation of the routing methodologies for three different RMINs. The sample RMINs are relatively small since the neural network router in its present form will only be competitive for small RMINs. The simulations show that the three routing schemes perform similarly for the three sample RMINs. Other criteria that will be touched upon are the speed and the resource utilization of each routing methodology and the pros and cons of each approach will be discussed. The results suggest that neural network routers may be appropriate for some communication applications involving RMINs.

Routing in Random Multistage Interconnection Networks

The design and use of interconnection networks is a topic that has generated considerable interest for many years. The reason for this interest is the wide applicability of the field. Interconnection networks have been widely studied and used for telecommunications, parallel processing, and distributed computing.