Efficient scheduling for multi-stage coflows

Abstract

In data center networks (DCN), large scale flows produced by parallel computing frameworks form many coflows semantically. Most inter-coflow schedulers only focus on the remaining data of coflows and attempt to mimic Shortest Job First (SJF). However, a coflow may consist of multiple stages, where a coflow has different amounts of data to transmit. In this paper, we consider the Multi-stage Inter-Coflow Scheduling problem and try to give an efficient online scheduling scheme. We first explore a short-sighted algorithm, IAO, with the greedy strategy. This gives us an insight into utilizing the network resources. Based on that, we propose a far-sighted heuristic, MLBF, which schedules sub-coflows to occupy network bandwidth in turn. Furthermore, we remove the bijection assumption and propose a new practical heuristic, MPLBF. Through simulations in various network environments, we show that, compared to a state-of-the-art scheduler—Varys, a multi-stage aware scheduler can reduce the coflow completion time by up to 4.81\xa0$$\\times$$× even though it is short-sighted. Moreover, the far-sighted scheduler MLBF can improve the performance by nearly 7.95\xa0$$\\times$$× reduction. Last but not least, MPLBF can improve the performance by up to 8.03\xa0$$\\times$$× reduction.