Songze Li

Fundamental tradeoff between computation and communication in distributed computing

We introduce a general distributed computing framework, motivated by commonly used structures like MapReduce, and formulate an information-theoretic tradeoff between computation and communication in such a framework. We characterize the optimal tradeoff to within a constant factor, for all system parameters. In particular, we propose a coded scheme, namely “Coded MapReduce” (CMR), which creates and exploits coding opportunities in data shuffling for distributed computing, reducing the communication load by a factor that is linearly proportional to the computation load. We then prove a lower bound on the minimum communication load, and demonstrate that CMR achieves this lower bound to within a constant factor. This result reveals a fundamental connection between computation and communication in distributed computing - the two are inverse-linearly proportional to each other.

Coding for Distributed Fog Computing

Redundancy is abundant in fog networks (i.e., many computing and storage points) and grows linearly with network size. We demonstrate the transformational role of coding in fog computing for leveraging such redundancy to substantially reduce the bandwidth consumption and latency of computing. In particular, we discuss two recently proposed coding concepts, minimum bandwidth codes and minimum latency codes, and illustrate their impacts on fog computing. We also review a unified coding framework that includes the above two coding techniques as special cases, and enables a trade-off between computation latency and communication load to optimize system performance. At the end, we will discuss several open problems and future research directions.

A Scalable Framework for Wireless Distributed Computing

We consider a wireless distributed computing system, in which multiple mobile users, connected wirelessly through an access point, collaborate to perform a computation task. In particular, users communicate with each other via the access point to exchange their locally computed intermediate computation results, which is known as data shuffling. We propose a scalable framework for this system, in which the required communication bandwidth for data shuffling does not increase with the number of users in the network. The key idea is to utilize a particular repetitive pattern of placing the data set (thus a particular repetitive pattern of intermediate computations), in order to provide the coding opportunities at both the users and the access point, which reduce the required uplink communication bandwidth from users to the access point and the downlink communication bandwidth from access point to users by factors that grow linearly with the number of users. We also demonstrate that the proposed data set placement and coded shuffling schemes are optimal (i.e., achieve the minimum required shuffling load) for both a centralized setting and a decentralized setting, by developing tight information-theoretic lower bounds.

How to optimally allocate resources for coded distributed computing

To execute cloud computing tasks over a data center hosting hundreds of thousands of server nodes, it is natural to distribute computations across the nodes to take advantage of parallel processing. However, as we allocate more computing resources and further distribute the computations, a large amount of intermediate data must be moved between consecutive computation stages among the nodes, causing the communication load to become the bottleneck. In this paper, we study the optimal resource allocation in distributed computing, in order to minimize the total execution time accounting for the durations of both computation and communication phases. Particularly, we consider a general MapReduce-type framework, and focus on a recently proposed Coded Distributed Computing approach. For all values of problem parameters, we characterize the optimal number of servers that should be used for computing, provide the optimal placements of the Map and Reduce tasks, and propose an optimal coded data shuffling scheme. To prove the optimality of the proposed scheme, we first derive a matching information-theoretic converse on the execution time, then we prove that among all resource allocation schemes that achieve the minimum execution time, our proposed scheme uses the exactly least number of servers.

Coded Distributed Computing: Straggling Servers and Multistage Dataflows

In this paper, we first review the Coded Distributed Computing (CDC) framework, recently proposed to significantly slash the data shuffling load of distributed computing via coding, and then discuss the extension of the CDC techniques to cope with two major challenges in general distributed computing problems, namely the straggling servers and multistage computations. When faced with straggling servers in a distributed computing cluster, we describe a unified coding scheme that superimposes CDC with the Maximum-Distance-Separable (MDS) coding on computation tasks, which allows a flexible tradeoff between computation latency and communication load. On the other hand, for a general multistage computation task expressed as a directed acyclic graph (DAG), we propose a coded framework that given the load of computation on each vertex of the DAG, applies the generalized CDC scheme individually on each vertex to minimize the communication load.

Jointly cooperative decode-and-forward relaying for secondary spectrum access

We propose a two-phase protocol based on cooperative decode-and-forward relaying for a secondary system to achieve spectrum access along with a primary system. The primary and secondary systems comprise of a transmitter-receiver pair, PT-PR and ST-SR, respectively. In the first transmission phase, PT transmits the primary signal to PR, which is also received by ST and SR, where it is decoded. At ST, the primary signal is regenerated and linearly combined with the secondary signal by assigning fractions alpha and (1 - alpha) of the available power to the primary and secondary signals respectively. This combined signal is then broadcasted by ST in the second transmission phase. We show that as long as ST is located within a critical radius from PT, there exists a threshold value for alpha above which the outage probability of the primary system will be equal to or lower than the case without spectrum sharing. We also determine the outage probability achieved by the secondary system. Theoretical and simulation results confirm the efficiency of the proposed spectrum sharing scheme.

Coded distributed computing: Fundamental limits and practical challenges

In this paper, we demonstrate a coded computing framework, named Coded Distributed Computing (CDC), which optimally trades extra computation resources for communication bandwidth in a MapReduce-type distributed computing environment. We also empirically illustrate the practical impact of CDC by analyzing the performance of a distributed sorting algorithm, named CodedTeraSort, which was developed by integrating the coding principle of CDC into the Hadoop benchmark TeraSort. Experiment results illustrate 1.97×–3.39 × speedup using CodedTeraSort, compared with TeraSort, for typical settings of interest. In the end, we review some of the open problems and future directions.

Communication-aware computing for edge processing

We consider a mobile edge computing problem, in which mobile users offload their computation tasks to computing nodes (e.g., base stations) at the network edge. The edge nodes compute the requested functions and communicate the computed results to the users via wireless links. For this problem, we propose a Universal Coded Edge Computing (UCEC) scheme for linear functions to simultaneously minimize the load of computation at the edge nodes, and maximize the physical-layer communication efficiency towards the mobile users. In the proposed UCEC scheme, edge nodes create coded inputs of the users, from which they compute coded output results. Then, the edge nodes utilize the computed coded results to create communication messages that zero-force all the interference signals over the air at each user. Specifically, the proposed scheme is universal since the coded computations performed at the edge nodes are oblivious of the channel states during the communication process from the edge nodes to the users.

Poster Abstract: A Scalable Coded Computing Framework for Edge-Facilitated Wireless Distributed Computing

We propose a scalable coded distributed computing framework for wireless distributed computing over a cluster of mobile users, in which the data shuffling across users are performed through an access point at the edge of the network. The proposed framework achieves a constant shuffling load that is independent of the number of participating users. The key idea is to utilize a particular repetitive structure of computation assignments at the users, in order to provide coding opportunities that reduce the shuffling load by a factor that grows linearly with the number of users.

Power allocation for Gaussian multiple access channel with noisy cooperative links

In this paper, a new coding scheme for the multiple access channel (MAC) with noisy cooperative links is proposed. The cooperation cost is modelled by powers spent on exchanging common information at transmitters. The optimal power allocation policy is derived to explore the tradeoff between cooperation and transmission. For some important cases, optimal power allocation that maximizes weighted sum rate, is found analytically. The sufficient and necessary condition for which the sum and the individual rates are simultaneously maximized, is identified. Analytical and numerical results suggest that the transmitter, whose power budget is dominated by that of the other, acts purely as a relay. The cooperation gain becomes more significant when the difference between the power budgets is smaller.