Fangfei Chen

Joint scheduling of processing and Shuffle phases in MapReduce systems

MapReduce has emerged as an important paradigm for processing data in large data centers. MapReduce is a three phase algorithm comprising of Map, Shuffle and Reduce phases. Due to its widespread deployment, there have been several recent papers outlining practical schemes to improve the performance of MapReduce systems. All these efforts focus on one of the three phases to obtain performance improvement. In this paper, we consider the problem of jointly scheduling all three phases of the MapReduce process with a view of understanding the theoretical complexity of the joint scheduling and working towards practical heuristics for scheduling the tasks. We give guaranteed approximation algorithms and outline several heuristics to solve the joint scheduling problem.

Intra-cloud lightning: Building CDNs in the cloud

Content distribution networks (CDNs) using storage clouds have recently started to emerge. Compared to traditional CDNs, storage cloud-based CDNs have the advantage of cost effectively offering hosting services to Web content providers without owning infrastructure. However, existing work on replica placement in CDNs does not readily apply in the cloud. In this paper, we investigated the joint problem of building distribution paths and placing Web server replicas in cloud CDNs to minimize the cost incurred on the CDN providers while satisfying QoS requirements for user requests. We formulate the cost optimization problem with accurate cost models and QoS requirements and show that the monthly cost can be as low as 2.62 US Dollars for a small Web site. We develop a suite of offline, online-static and online-dynamic heuristic algorithms that take as input network topology and work load information such as user location and request rates. We then evaluate the heuristics via Web trace-based simulation, and show that our heuristics behave very close to optimal under various network conditions.

Demo: A distributed architecture for heterogeneous multi sensor-task allocation

To complement our paper “A Distributed Architecture For Heterogeneous Multi-Sensor Task Allocation” published in DCOSS 2011 conference proceedings, we will demonstrate the current implementation of our architecture for Multi-Sensor Task Allocation (MSTA) [2] shown in Figure 1. This consists of a prototype interface for task submission, and a simulated sensor network serving other mobile users on the field competing for the same sensing resources. The prototype interface is implemented on two different mobile devices (iPhone & iPad1), dynamically interacting with a heterogeneous sensor network simulated in Java on top of the REPAST Symphony2 discrete time simulation environment. A video of the demo is available at http://www.youtube.com/watch?v=QzrpKRhGFRU.

Who, When, Where: Timeslot Assignment to Mobile Clients

We consider variations of a problem in which data must be delivered to mobile clients en route, as they travel toward their destinations. The data can only be delivered to the mobile clients as they pass within range of wireless base stations. Example scenarios include the delivery of building maps to firefighters responding to multiple alarms. We cast this scenario as a parallel-machine scheduling problem with the little-studied property that jobs may have different release times and deadlines when assigned to different machines. We present new algorithms and also adapt existing algorithms, for both online and offline settings. We evaluate these algorithms on a variety of problem instance types, using both synthetic and real-world data, including several geographical scenarios, and show that our algorithms produce schedules achieving near-optimal throughput.

Resource Allocation with Stochastic Demands

Resources in modern computer systems include not only CPU, but also memory, hard disk, bandwidth, etc. To serve multiple users simultaneously, we need to satisfy their requirements in all resource dimensions. Meanwhile, their demands follow a certain distribution and may change over time. Our goal is then to admit as many users as possible to the system without violating the resource capacity more often than a predefined overflow probability. In this paper, we study the problem of allocating multiple resources among a group of users/tasks with stochastic demands. We model it as a stochastic multi-dimensional knapsack problem. We extend and apply the concept of effective bandwidth in order to solve this problem efficiently. Via numerical experiments, we show that our algorithms achieve near-optimal performance with specified overflow probability.

A system architecture for exploiting mission information requirement and resource allocation

In a military scenario, commanders need to determine what kinds of information will help them execute missions. The amount of information available to support each mission is constrained by the availability of information assets. For example, there may be limits on the numbers of sensors that can be deployed to cover a certain area, and limits on the bandwidth available to collect data from those sensors for processing. Therefore, options for satisfying information requirements should take into consideration constraints on the underlying information assets, which in certain cases could simultaneously support multiple missions. In this paper, we propose a system architecture for modeling missions and allocating information assets among them. We model a mission as a graph of tasks with temporal and probabilistic relations. Each task requires some information provided by the information assets. Our system suggests which information assets should be allocated among missions. Missions are compatible with each other if their needs do not exceed the limits of the information assets; otherwise, feedback is sent to the commander indicating information requirements need to be adjusted. The decision loop will eventually converge and the utilization of the resources is maximized.

Proactive Data Dissemination to Mission Sites

In many situations it is important to deliver information to personnel as they work in the field. We consider such a specialized content distribution application in wireless mesh networks. When a new mission arrives-for example, when an alarm for a fire is reported-data is pushed to storage nodes at the mission site where it may be retrieved locally by responding personnel (e.g., police, firefighters, paramedics, government officials, and the media). It is important that information is available at low latency, when requested or pulled by the personnel. The total latency experienced will be a combination of the push delay (if the personnel arrive at the mission site before all the data can be pushed), and the pull delay. Each delay component will in turn be a function of 1) the hop distance traveled by the data when pushed or pulled and 2) the congestion on the links. In this paper, we define algorithms and protocols that trade-off the push and pull latencies depending on the type of application. Our goal is to choose a storage node assignment minimizing the total latency-based cost. We start with a simple model in which cost is a function of distance, and then extend the model explicitly taking congestion into account. Since the problem is NP-hard to approximate, our focus is on developing efficient algorithms and distributed protocols that can be easily deployed in wireless mesh networks. In NS2 simulations, we find that our heuristic algorithms achieve on average a cost within at most 15% of the optimum.

Multiple Backhaul Mobile Access Router: Design and Experimentation

The multiple backhaul mobile access router aims to provide high capacity and high performance Internet access for emerging mobile wireless applications. In this paper we describe the framework and implementation of a modular mobile access router system. A set of backhaul interface monitoring APIs are provided to support flexible handover policies. We present the experimental results to validate the performance of our mobile access router and illustrate tradeoffs when designing handover policies.

Convergecast with aggregatable data classes

Data-gathering or convergecast problems have traditionally been studied in two combinations of settings: one-shot scheduling of data items with no aggregation, and periodic scheduling of data items with full aggregation meaning that any number of unit-size data items can, if available, be aggregated into a single (unit-size) data item (e.g., by summing or averaging values). In this paper, we extend beyond these problem settings in two ways. First, we study a) one-shot throughput maximization in settings with aggregation and b) periodic scheduling in settings without aggregation. Second, we generalize the notion of aggregatability in both one-shot and periodic scheduling beyond the binary choice of either all sets of items being aggregatable or none being so. Modeling the presence of multiple semantic data types (e.g., target counts to be summed and temperature readings to be averaged), we partition data items into classes, whereby items are aggregatable if they belong to the same class, in both periodic and non-periodic settings. For these two problems we provide guaranteed approximations and heuristics, for a variety of general and special cases. We then evaluate the algorithms in a systematic simulation study, both under the conditions in which our provable guarantees apply and in more general settings, where we find the algorithms continue to perform well on typical problem inputs.

Who, when, where: Timeslot assignment to mobile clients

We consider variations of a problem in which data must be delivered to mobile clients en route, as they travel toward their destinations. The data can only be delivered to the mobile clients as they pass within range of wireless base stations. Example scenarios include the delivery of building maps to firefighters responding to multiple alarms. We cast this scenario as a parallel-machine scheduling problem with the little-studied property that jobs may have different release times and deadlines when assigned to different machines. We present new algorithms and also adapt existing algorithms, for both online and offline settings. We evaluate these algorithms on a variety of problem instance types, using both synthetic and real-world data, including several geographical scenarios, and show that our algorithms produce schedules achieving near-optimal throughput.