Jay Lepreau

An integrated experimental environment for distributed systems and networks

Three experimental environments traditionally support network and distributed systems research: network emulators, network simulators, and live networks. The continued use of multiple approaches highlights both the value and inadequacy of each. Netbed, a descendant of Emulab, provides an experimentation facility that integrates these approaches, allowing researchers to configure and access networks composed of emulated, simulated, and wide-area nodes and links. Netbeds primary goals are ease of use, control, and realism, achieved through consistent use of virtualization and abstraction.By providing operating system-like services, such as resource allocation and scheduling, and by virtualizing heterogeneous resources, Netbed acts as a virtual machine for network experimentation. This paper presents Netbeds overall design and implementation and demonstrates its ability to improve experimental automation and efficiency. These, in turn, lead to new methods of experimentation, including automated parameter-space studies within emulation and straightforward comparisons of simulated, emulated, and wide-area scenarios.

The Flux OSKit: a substrate for kernel and language research

Implementing new operating systems is tedious, costly, and often impractical except for large projects. The Flux OSKit addresses this problem in a novel way by providing clean, well-documented OS components designed to be reused in a wide variety of other environments, rather than defining a new OS structure. The OSKit uses unconventional techniques to maximize its usefulness, such as intentionally exposing implementation details and platform-specific facilities. Further, the OSKit demonstrates a technique that allows unmodified code from existing mature operating systems to beincorporated quickly and updated regularly, by wrapping it with a small amount of carefully designed “glue” code to isolate it s dependencies and export well-defined interfaces. The OSKit uses this technique to incorporate over 230,000 lines of stable code including device drivers, file systems, and network protocols. Ourexperience demonstrates that this approach to component software structure and reuse has a surprisingly large impact in the OS implementation domain. Four real-world examples show how the OSKit is catalyzing research and development in operating systems and programming languages.

A solver for the network testbed mapping problem

Network experiments of many types, especially emulation, require the ability to map virtual resources requested by an experimenter onto available physical resources. These resources include hosts, routers, switches, and the links that connect them. Experimenter requests, such as nodes with special hardware or software, must be satisfied, and bottleneck links and other scarce resources in the physical topology should be conserved when physical resources are shared. In the face of these constraints, this mapping becomes an NP-hard problem. Yet, in order to prevent mapping time from becoming a serious hindrance to experimentation, this process cannot consume an excessive amount of time.In this paper, we explore this problem, which we call the network testbed mapping problem.We describe the interesting challenges that characterize it, and explore its applications to emulation and other spaces, such as distributed simulation. We present the design, implementation, and evaluation of a solver for this problem, which is in production use on the Netbed shared network testbed. Our solver builds on simulated annealing to find very good solutions in a few seconds for our historical workload, and scales gracefully on large well-connected synthetic topologies.

Microkernels meet recursive virtual machines

This paper describes a novel approach to providingmodular and extensible operating system functionality and encapsulated environments based on a synthesis of microkernel and virtual machine concepts. We have developed a software-based virtualizable architecture called Fluke that allows recursive virtual machines (virtual machines running on other virtual machines) to be implemented efficiently by a microkernel running on generic hardware. A complete virtual machine interface is provided at each level; efficiency derives from needing to implement only new functionality at each level. This infrastructure allows common OS functionality, such as process management, demand paging, fault tolerance, and debugging support, to be provided by cleanly modularized, independent, stackable virtual machine monitors, implemented as user processes. It can also provide uncommon or unique OS features, including the above features specialized for particular applications’ needs, virtual machines transparently distributed cross-node, or security monitors that allow arbitrary untrusted binaries to be executed safely. Our prototype implementation of this model indicates that it is practical to modularize operating systems this way. Some types of virtual machine layers impose almost no overhead at all, while others impose some overhead (typically 0–35%), but only on certain classes of applications.

Mobile Emulab: A Robotic Wireless and Sensor Network Testbed

Simulation has been the dominant research method- ology in wireless and sensor networking. When mobility is added, real-world experimentation is especially rare. However, it is becoming clear that simulation models do not sufficiently capture radio and sensor irregularity in a complex, real-world environment, especially indoors. Unfortunately, the high labor and equipment costs of truly mobile experimental infrastructure present high barriers to such experimentation. We describe our experience in creating a testbed to lower those barriers. We have extended the Emulab network testbed software to provide the first remotely-accessible mobile wireless and sensor testbed. Robots carry motes and single board computers through a fixed indoor field of sensor-equipped motes, all running the users selected software. In real-time, interactively or driven by a script, remote users can position the robots, control all the computers and network interfaces, run arbitrary programs, and log data. Our mobile testbed provides simple path planning, a vision-based tracking system accurate to 1 cm, live maps, and webcams. Precise positioning and automation allow quick and painless evaluation of location and mobility effects on wireless protocols, location algorithms, and sensor-driven applications. The system is robust enough that it is deployed for public use. We present the design and implementation of our mobile testbed, evaluate key aspects of its performance, and describe a few experiments demonstrating its generality and power.

An OS interface for active routers

This paper describes an operating system (OS) interface for active routers. This interface allows code loaded into active routers to access the routers memory, communication, and computational resources on behalf of different packet flows. In addition to motivating and describing the interface, the paper also reports our experiences implementing the interface in three different OS environments: Scout, the OSKit, and the esokernel.

Evolving real-time systems using hierarchical scheduling and concurrency analysis

We have developed a new way to look at real-time and embedded software: as a collection of execution environments created by a hierarchy of schedulers. Common schedulers include those than run interrupts, bottom-half handlers, threads, and events. We have created algorithms for deriving response times, scheduling overheads, and blocking terms for tasks in systems containing multiple execution environments. We have also created task scheduler logic, a formalism that permits checking systems for race conditions and other errors. Concurrency analysis of low-level software is challenging because there are typically several kinds of locks, such as thread mutexes and disabling interrupts, and groups of cooperating tasks may need to acquire some, all or none of the available types of locks to create correct software. Our high-level goal is to create systems that are evolvable: they are easier to modify in response to changing requirements than are systems created using traditional techniques. We have applied our approach to two case studies in evolving software for networked sensor nodes.

Lowering the barrier to wireless and mobile experimentation

The success of ns highlights the importance of an infrastructure that enables efficient experimentation. Similarly, Netbeds automatic configuration and control of emulated and live network environments minimizes the effort spent configuring and running experiments. Learning from the evolution of these systems, in this paper we argue that a live wireless and mobile experimental facility focusing on ease of use and accessibility will not only greatly lower the barrier to research in these areas, but that the primary technical challenges can be overcome.The flexibility of Netbeds common abstractions for diverse node and link types has enabled its development from strictly an emulation platform to one that integrates simulation and live network experimentation. It can be further extended to incorporate wireless and mobile devices. To reduce the tedium of wireless and mobile experimentation, we propose automatically allocating and mapping a subset of a dense mesh of devices to match a specified network topology. To achieve low-overhead, coarse repeatability for mobile experiments, we outline how to leverage the predictability of passive couriers, such as PDA-equipped students and PC-equipped busses.

Nested Java processes: OS structure for mobile code

The majority of work on protection in single-language mobile code environments focuses on information security issues and depends on the language environment for solutions to the problems of resource management and process isolation. We believe that what is needed in these environments are not ad-hoc or incremental changes but a coherent approach to security, failure isolation, and resource management. Protection, separation, and control of the resources used by mutually untrusting components, applets, applications, or agents are exactly the same problems faced by multi-user operating systems. We believe that real solutions will come only if an OS model is uniformly applied to these environments. We present Alta, our prototype Java-based system patterned on Fluke, a highly structured, hardware-based OS, and report on its features appropriate to mobile code. 1 Operating System Model Required In the last European SIGOPS Workshop, our paper [17] argued that the local operating system is an essential foundation for global applications. We described the many demands that a reasonably well functioning distributed system places on the local OS, and particularly emphasized end-system security in the widespread presence of mobile code. The focus of that paper was on making the case for the importance of the local OS, and outlining an appropriate OS for that environment: the Fluke [10] operating system, an OS based on a recursive virtual machine model, analogous to the Cambridge CAP Computer [30], but implemented by a microkernel instead of special hardware. In this paper we assume that the importance of the local This research was supported in part by the Defense Advanced Research Projects Agency, monitored by the Department of the Army under contract number DABT63–94–C–0058, and the Air Force Research Laboratory, Rome Research Site, USAF, under agreement number F30602–96–2–0269. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation hereon. OS to distributed applications is evident. From that base, we endeavor to make four points concerning platforms for mixed trust components and mobile code: (i) A coherent, structured approach is required, driven by a full-blown OS model; language-level patches are not enough. (ii) Existing security-oriented approaches fall short in resource management. (iii) Applying an OS model is feasible, based upon our initial experiences with Alta. (iv) Alta provides features useful for mobile code, including hierarchical resource management and flexible object sharing. 1.1 Application Scenario In 1997 MCI developed and distributed its Denial of Service Tracker (DoSTracker) [19], after getting their router vendor to add the required interfaces and code to the routers. DoSTracker works as follows. Many denial of service attacks involve generating packets that spoof the IP address of the victim’s host. For example, fabricating broadcast packets will generate a storm of replies to the “sender.” When a customer reports an attack on a particular host, their ISP runs DoSTracker on a machine connected to the victim’s router, giving it the victim’s IP address. DoSTracker hops from router to router, following spoofed broadcast packets “upstream” to the actual source. Problems arise when the path leads into another Internet carrier’s hosts—a different administrative and technical domain—whose routers may well not support the required interfaces. Similar hard to predict problems arise constantly in network management, and solutions are difficult to deploy quickly, and almost impossible to standardize. A first step to providing network administrators with a solution to these problems might let them run mobile programs on the routers. This, of course, is one example of an active network [28]. One need not commit to the aggressive vision of active networks—code in any packet—to appreciate the value of supporting mobile code in routers. Network management is an application domain that could greatly profit from mobile code and dynamic composition of mobile components. However, along with the solutions proffered by mobile code there must be strong security guarantees and flexible, hierarchical resource management. Consider the following realistic Internet-wide scenario of hierarchical trust and proportional share resource management. MCI reserves 80% of the resources in each of its routers for “real work” (i.e., forwarding packets). The other 20% is available on demand for management functions (such as DoSTracker), mobile code, or agents. 50% of that (i.e., 10% of the total) is reserved for MCI’s own management routines, with the rest available to its customers. However, all customers are not equal, so MCI allocates 50% of that 10% to the 20-odd long-haul Internet carriers, such as Digex1 or AT&T, and the other 50% to other customers (e.g., ISPs). The 5% allocated to the long-haul Internet carriers could again be split up equally among the carriers—effectively each internet carrier owns a modest 0:25% of every other carrier’s available bandwidth. Digex manages its portion (on any carrier), allocating half to trusted (to Digex) requests from its own network management, and the other half to Digex customers. See Figure 1. Clearly, a hierarchical, extensible resource management model would provide the ability to recursively refine system allocation. Additionally, a stringent security infrastructure to authenticate and manage the mobile agents in such a system is required. Relative Processor Allocation Forwarding Managment

Upgrading transport protocols using untrusted mobile code

In this paper, we present STP, a system in which communicating end hosts use untrusted mobile code to remotely upgrade each other with the transport protocols that they use to communicate. New transport protocols are written in a type-safe version of C, distributed out-of-band, and run in-kernel. Communicating peers select a transport protocol to use as part of a TCP-like connection setup handshake that is backwards-compatible with TCP and incurs minimum connection setup latency. New transports can be invoked by unmodified applications. By providing a late binding of protocols to hosts, STP removes many of the delays and constraints that are otherwise commonplace when upgrading the transport protocols deployed on the Internet. STP is simultaneously able to provide a high level of security and performance. It allows each host to protect itself from untrusted transport code and to ensure that this code does not harm other network users by sending significantly faster than a compliant TCP. It runs untrusted code with low enough overhead that new transport protocols can sustain near gigabit rates on commodity hardware. We believe that these properties, plus compatibility with existing applications and transports, complete the features that are needed to make STP useful in practice.