Greg Eisenhauer

Open metadata formats: efficient XML-based communication for high performance computing

High-performance computing faces considerable change as the Internet and the Grid mature. Applications that once were tightly-coupled and monolithic are now decentralized, with collaborating components spread across diverse computational elements. Such distributed systems most commonly communicate through the exchange of structured data. Definition and translation of metadata is incorporated in all systems that exchange structured data. We observe that the manipulation of this metadata can be decomposed into three separate steps: discovery, binding of program objects to the metadata, and marshaling of data to and from wire formats. We have designed a method of representing message formats in XML, using datatypes available in the XML Schema specification. We have implemented a tool, XMIT, that uses such metadata and exploits this decomposition in order to provide flexible run-time metadata definition facilities for an efficient binary communication mechanism. We also demonstrate that the use of XMIT makes possible such flexibility at little performance cost.

A middleware toolkit for client-initiated service specialization

As the Internet matures, streaming data services are taking an increasingly important place alongside traditional HTTP transactions. The need to dynamically adjust the delivery of such services to changes in available network and processing resources has spawned substantial research on application-specific methods for dynamic adaptation, including video and audio streaming applications. Such adaptation techniques are well developed, but they are also highly specialized, with the client (receiver) and server (sender) implementing well-defined protocols that exploit content-specific stream properties. This paper describes our efforts to bring the benefits of such content-aware, application-level service adaptation to all types of streaming data and to do so in a manner that is efficient and flexible. Our contribution in this domain is ECho, a high-performance event-delivery middleware system. EChos basic functionality provides efficient binary transmission of event data with unique features that support dynamic data-type discovery and service evolution. EChos contribution to data stream adaptation is in the mechanisms it provides for its clients to customize their data flows through type-safe dynamic server extension.

Active Streams - an approach to adaptive distributed systems

Summary form only given. An increasing number of distributed applications aim to provide services to users by interacting with a correspondingly growing set of data-intensive network services. To support such requirements, we believe that new services need to be customizable, applications need to be dynamically extensible, and both applications and services need to be able to adapt to variations in resource availability and demand. A comprehensive approach to building new distributed applications can facilitate this by considering the contents of the information flowing across the application and its services and by adopting a component-based model to application/service programming. It should provide for dynamic adaptation at multiple levels and points in the underlying platform; and, since the mapping of components to resources in dynamic environment is too complicated, it should relieve programmers of this task. We propose Active Streams, a middleware approach and its associated framework for building distributed applications and services that exhibit these characteristics.

A machine independent interface for lightweight threads

Recently, lightweight thread libraries have become a common entity to support concurrent programming on shared memory multiprocessors. However, the disparity between primitives offered by operating systems creates a challenge for those who wish to create portable lightweight thread packages. What should be the interface between the machine-independent and machine-dependent parts of the thread library? We have implemented a portable lightweight thread library on top of Unix on a KSR-1 supercomputer, BBN Butterfly multiprocessor, SGI multiprocessor, Sequent multiprocessor and Sun 3/4 family of uniprocessors. This paper first compares the nature and performance of the OS primitives offered by these machines. We then present procedure-level abstraction that is efficiently implementable on all the architectures and is a sufficient base upon which a user-level thread package can be built.

Service morphing: integrated system- and application-level service adaptation in autonomic systems

Service morphing is a set of techniques used to continuously meet an applications quality of service (QoS) needs, in the presence of run-time variations in service locations, platform capabilities, or end-user needs. These techniques provide high levels of flexibility in how, when, and where necessary processing and communication actions are performed. Lightweight middleware supports flexibility by permitting end-users to subscribe to information channels of interest to them whenever they desire, and then apply exactly the processing to such information they require. New compiler and binary code generation techniques dynamically generate, deploy, and specialize code in order to match current user needs to available platform resources. Finally, to deal with run-time changes in resource availability, kernel-level resource management mechanisms are associated with user-level middleware. Such associations range from loosely coupled, where kernel-level resource management monitors and occasionally responds to userlevel events, to tightly coupled, where kernel-level mechanisms import, export, and use performance and control attributes in conjunction with each resource-relevant userlevel event.

KECho - Event Communication for Distributed Kernel Services

Event services have received increased attention as scalable tools for the composition of large-scale, distributed systems, as evidenced by their successful deployment in interactive multimedia applications and scientific collaborative tools. This paper introduces KECho, a kernelbased event service aimed at supporting the coordination among multiple kernel services in distributed systems, typically to provide applications using these services with certain levels of Quality of Service (QoS). The publish/subscribe communication supported by KECho permits components of remote kernels as well as applications to coordinate their operation. The target group of such a kernel-based event service is the rapidly increasing number of extensions that are being added to existing operating systems and are intended to support the Quality of Service and real-time requirements of distributed and embedded applications.

Event Services in High Performance Systems

The Internet and the Grid are changing the face of high performance computing. Rather than tightly-coupled SPMD-style components running in a single cluster, on a parallel machine, or even on the Internet programmed in MPI, applications are evolving into sets of cooperating components scattered across diverse computational elements. These components may run on different operating systems and hardware platforms and may be written by different organizations in different languages. Complete “applications” are constructed by assembling these components in a plug-and-play fashion. This new vision for high performance computing demands features and characteristics not easily provided by traditional high-performance communications middleware. In response to these needs, we have developed ECho, a high-performance event-delivery middleware that meets the new demands of the Grid environment. ECho provides efficient binary transmission of event data with unique features that support data-type discovery and enterprise-scale application evolution. We present measurements detailing EChos performance to show that ECho significantly outperforms other systems intended to provide this functionality and provides throughput and latency comparable to the most efficient middleware infrastructures available.

Open metadata formats: efficient XML-based communication for heterogeneous distributed systems

The definition and translation of metadata is incorporated in all systems that exchange structured data. We observe that the manipulation of this metadata can be decomposed into three separate steps: discovery, binding of program objects to the metadata, and marshaling of data to and from wire formats. We have designed a method of representing message formats in XML, using data types that are available in the XML schema specification. We have implemented a tool called xml2wire that uses such metadata and exploits this decomposition in order to provide flexible metadata definition facilities for an efficient binary communications mechanism. We also observe that the use of xml2wire makes possible such flexibility without intolerable performance effects.

Active brokers and their runtime deployment in the ECho/JECho distributed event systems

This paper introduces active brokers and third-party derivation, which is the basic programming construct for run-time remote broker deployment, in the ECho/JECho distributed event systems. We describe its implementation in the JECho system and give examples of using it in ECho/JECho distributed event systems. In particular, we describe the use of third-party derivation in supporting the scalability of content-based event delivery. Specifically, third-party derivation is used both to dynamically construct content-based event distribution trees and to offload potentially expensive client-specific event routing/processing by the run-time creation of remote brokers. Our preliminary benchmark results demonstrate the significant benefits of using third-party derivation.