Malte Schwarzkopf

Omega: flexible, scalable schedulers for large compute clusters

Increasing scale and the need for rapid response to changing requirements are hard to meet with current monolithic cluster scheduler architectures. This restricts the rate at which new features can be deployed, decreases efficiency and utilization, and will eventually limit cluster growth. We present a novel approach to address these needs using parallelism, shared state, and lock-free optimistic concurrency control. We compare this approach to existing cluster scheduler designs, evaluate how much interference between schedulers occurs and how much it matters in practice, present some techniques to alleviate it, and finally discuss a use case highlighting the advantages of our approach -- all driven by real-life Google production workloads.

Firmament: fast, centralized cluster scheduling at scale

Centralized datacenter schedulers can make high-quality placement decisions when scheduling tasks in a cluster. Today, however, high-quality placements come at the cost of high latency at scale, which degrades response time for interactive tasks and reduces cluster utilization. This paper describes Firmament, a centralized scheduler that scales to over ten thousand machines at sub-second placement latency even though it continuously reschedules all tasks via a min-cost max-flow (MCMF) optimization. Firmament achieves low latency by using multiple MCMF algorithms, by solving the problem incrementally, and via problem-specific optimizations. Experiments with a Google workload trace from a 12,500-machine cluster show that Firmament improves placement latency by 20× over Quincy [22], a prior centralized scheduler using the same MCMF optimization. Moreover, even though Firmament is centralized, it matches the placement latency of distributed schedulers for workloads of short tasks. Finally, Firmament exceeds the placement quality of four widely-used centralized and distributed schedulers on a real-world cluster, and hence improves batch task response time by 6×.

Using Dust Clouds to Enhance Anonymous Communication

Cloud computing platforms, such as Amazon EC2 [1], enable customers to lease several virtual machines (VMs) on a per-hour basis. The customer can now obtain a dynamic and diverse collection of machines spread across the world. In this paper we consider how this aspect of cloud computing can facilitate anonymous communications over untrusted networks such as the Internet, and discuss some of the challenges that arise as a result.

Non-volatile storage

For the entire careers of most practicing computer scientists, a fundamental observation has consistently held true: CPUs are significantly more performant and more expensive than I/O devices. The fact that CPUs can process data at extremely high rates, while simultaneously servicing multiple I/O devices, has had a sweeping impact on the design of both hardware and software for systems of all sizes, for pretty much as long as we’ve been building them.

New wine in old skins: the case for distributed operating systems in the data center

Since their heyday the 1980s, distributed operating systems---spanning multiple autonomous machines, yet appearing to the user as a single machine---have seen only moderate academic interest. This is a little surprising, since modern data centers might present an appealing environment for their deployment. In this position paper, we discuss what has changed since the community lost interest in them, and why, nonetheless, distributed OSes have yet to be considered for data centers. Finally, we argue that the distributed OS concept is worth a revisit, and outline the benefits to be had from it in the context of the modern data center.

R2D2: bufferless, switchless data center networks using commodity ethernet hardware

Modern data centers commonly run distributed applications that require low-latency communication, and whose performance is critical to service revenue. If as little as one machine in 10,000 is a latency outlier, around 18% of requests will experience high latency. The sacrifice of latency determinism for bandwidth, however, is not an inevitable one. In our R2D2 architecture, we conceptually split the data centre network into an unbuffered, unswitched low-latency network (LLNet) and a deeply buffered bandwidth centric network (BBNet). Through explicitly scheduling network multiplexing in software, our prototype implementation achieves 99.995% and 99.999% messaging latencies of 35us and 75us respectively for 1514-byte packets on a fully loaded network. Furthermore, we show that it is possible to merge the conceptually separate LLNet and BBNet networks onto the same physical infrastructure using commodity switched Ethernet hardware.

Evaluating end-to-end optimization for data analytics applications in weld

Modern analytics applications use a diverse mix of libraries and functions. Unfortunately, there is no optimization across these libraries, resulting in performance penalties as high as an order of magnitude in many applications. To address this problem, we proposed Weld, a common runtime for existing data analytics libraries that performs key physical optimizations such as pipelining under existing, imperative library APIs. In this work, we further develop the Weld vision by designing an automatic adaptive optimizer for Weld applications, and evaluating its impact on realistic data science workloads. Our optimizer eliminates multiple forms of overhead that arise when composing imperative libraries like Pandas and NumPy, and uses lightweight measurements to make data-dependent decisions at run-time in ad-hoc workloads where no statistics are available, with sub-second overhead. We also evaluate which optimizations have the largest impact in practice and whether Weld can be integrated into libraries incrementally. Our results are promising: using our optimizer, Weld accelerates data science workloads by up to 23X on one thread and 80X on eight threads, and its adaptive optimizations provide up to a 3.75X speedup over rule-based optimization. Moreover, Weld provides benefits if even just 4--5 operators in a library are ported to use it. Our results show that common runtime designs like Weld may be a viable approach to accelerate analytics.

Research for practice: cluster scheduling for datacenters

Expert-curated guides to the best of CS research.

Cluster Scheduling for Data Centers

This installment of Research for Practice features a curated selection from Malte Schwarzkopf, who takes us on a tour of distributed cluster scheduling, from research to practice, and back again. With the rise of elastic compute resources, cluster management has become an increasingly hot topic in systems R&D, and a number of competing cluster managers including Kubernetes, Mesos, and Docker are currently jockeying for the crown in this space.

Using Dust Clouds to Enhance Anonymous Communication (Transcript of Discussion)

This presentation is about dust clouds in third party anonymity, and the fundamental technology that we’re building on is mix networks, which I assume most of you are familiar with. The idea is Alice wants to anonymously communicate with Bob. She does not want to be anonymous to Bob, but wants to be anonymous to some evil Eve that’s observing the network, and to do that she sends her data to a mix network, it goes into hiding from the ingress point to the egress point, and then comes up to Bob, and Eve observing parts of the mix network can’t tell what’s being said or who’s talking to whom, because it all lies encrypted in the network. However, that does not solve the problem if Eve is able to look at both the ingress and egress point of the network, then she can still see what’s going on, so if some evil overlord has a view of the entire Internet (a global passive adversary), we can’t be using a mix network. That’s the basic setting we’re looking at, and the specific mix network that we are concerned with is Tor; we think that our strategy would pretty much work with any mix network, but Tor is the example that we used in the paper.