Ganesh Ananthanarayanan

Scarlett: coping with skewed content popularity in mapreduce clusters

To improve data availability and resilience MapReduce frameworks use file systems that replicate data uniformly. However, analysis of job logs from a large production cluster shows wide disparity in data popularity. Machines and racks storing popular content become bottlenecks; thereby increasing the completion times of jobs accessing this data even when there are machines with spare cycles in the cluster. To address this problem, we present Scarlett, a system that replicates blocks based on their popularity. By accurately predicting file popularity and working within hard bounds on additional storage, Scarlett causes minimal interference to running jobs. Trace driven simulations and experiments in two popular MapReduce frameworks (Hadoop, Dryad) show that Scarlett effectively alleviates hotspots and can speed up jobs by 20.2%.

Blue-Fi: enhancing Wi-Fi performance using bluetooth signals

Mobile devices are increasingly equipped with multiple network interfaces with complementary characteristics. In particular, the Wi-Fi interface has high throughput and transfer power efficiency, but its idle power consumption is prohibitive. In this paper we present, Blue-Fi, a sytem that predicts the availability of the Wi-Fi connectivity by using a combination of bluetooth contact-patterns and cell-tower information. This allows the device to intelligently switch the Wi-Fi interface on only when there is Wi-Fi connectivity available, thus avoiding the long periods in idle state and significantly reducing the the number of scans for discovery. Our prediction results on traces collected from real users show an average coverage of 94% and an average accuracy of 84%, a 47% accuracy improvement over pure cell-tower based prediction, and a 57% coverage improvement over the pure bluetooth based prediction. For our workload, Blue-Fi is up to 62% more energy efficient, which results in increasing our mobile devices lifetime by more than a day.

COMBINE: leveraging the power of wireless peers through collaborative downloading

Mobile devices are increasingly equipped with multiple network interfaces: Wireless Local Area Network (WLAN) interfaces for local connectivity and Wireless Wide Area Network (WWAN) interfaces for wide-area connectivity. The WWAN typically provides much wider coverage but much lower speeds than the WLAN. To address this dichotomy, we present COMBINE, a system for collaborative downloading wherein devices that are within WLAN range pool together their WWAN links, significantly increasing the effective speed available to them. COMBINE makes a number of novel contributions overprior work in this area, including: (a) a framework of incentives for collaboration that addresses several practical issues including the unification of monetary and energy costs, and on-the-fly estimation of the energy cost of communication in a system in operation; (b) a protocol for collaborative group formation and workload distribution that is energy efficient and adaptive to fluctuations in network conditions; and (c) an application-level striping procedure that eases deployment by avoiding the need for special-purpose proxies in the infrastructure. We present experimental results based on the prototype we have implemented that showen couraging speeds-ups with COMBINE.

StarTrack: a framework for enabling track-based applications

Mobile devices are increasingly equipped with hardware and software services allowing them to determine their locations, but support for building location-aware applications remains rudimentary. This paper proposes tracks of location coordinates as a high-level abstraction for a new class of mobile applications including ride sharing, location-based collaboration, and health monitoring. Each track is a sequence of entries recording a persons time, location, and application-specific data. StarTrack provides applications with a comprehensive set of operations for recording, comparing, clustering and querying tracks. StarTrack can efficiently operate on thousands of tracks.

Hopper: Decentralized Speculation-aware Cluster Scheduling at Scale

As clusters continue to grow in size and complexity, providing scalable and predictable performance is an increasingly important challenge. A crucial roadblock to achieving predictable performance is stragglers, i.e., tasks that take significantly longer than expected to run. At this point, speculative execution has been widely adopted to mitigate the impact of stragglers. However, speculation mechanisms are designed and operated independently of job scheduling when, in fact, scheduling a speculative copy of a task has a direct impact on the resources available for other jobs. In this work, we present Hopper, a job scheduler that is speculation-aware, i.e., that integrates the tradeoffs associated with speculation into job scheduling decisions. We implement both centralized and decentralized prototypes of the Hopper scheduler and show that 50% (66%) improvements over state-of-the-art centralized (decentralized) schedulers and speculation strategies can be achieved through the coordination of scheduling and speculation.

Collaborative Downloading for Multi-homed Wireless Devices

Mobile devices are increasingly equipped with multiple network interfaces: Wireless Local Area Network (WLAN) interfaces for local connectivity and Wireless Wide Area Network (WWAN) interfaces for wide-area connectivity. The WWAN typically provides much wider coverage but much lower speeds than the WLAN. To address this dichotomy, we consider collaborative downloading among mobile devices in close proximity. We demonstrate the potential benefits of such an approach and discuss the many challenges to realizing it in practice: incentivizing cooperation by adequately compensating nodes, effecting such cooperation via an efficient protocol, and facilitating it with a suitable user interface. We present our current thinking on these as we design a collaborative downloading system called COMBINE.

True elasticity in multi-tenant data-intensive compute clusters

Data-intensive computing (DISC) frameworks scale by partitioning a job across a set of fault-tolerant tasks, then diffusing those tasks across large clusters. Multi-tenanted clusters must accommodate service-level objectives (SLO) in their resource model, often expressed as a maximum latency for allocating the desired set of resources to every job. When jobs are partitioned into tasks statically, a cluster cannot meet its SLOs while maintaining both high utilization and efficiency. Ideally, we want to give resources to jobs when they are free but would expect to reclaim them instantaneously when new jobs arrive, without losing work. DISC frameworks do not support such elasticity because interrupting running tasks incurs high overheads. Amoeba enables lightweight elasticity in DISC frameworks by identifying points at which running tasks of over-provisioned jobs can be safely exited, committing their outputs, and spawning new tasks for the remaining work. Effectively, tasks of DISC jobs are now sized dynamically in response to global resource scarcity or abundance. Simulation and deployment of our prototype shows that Amoeba speeds up jobs by 32% without compromising utilization or efficiency.

Wrangler: Predictable and Faster Jobs using Fewer Resources

Straggler tasks continue to be a major hurdle in achieving faster completion of data intensive applications running on modern data-processing frameworks. Existing straggler mitigation techniques are inefficient due to their reactive and replicative nature -- they rely on a wait-speculate-re-execute mechanism, thus leading to delayed straggler detection and inefficient resource utilization. Existing proactive techniques also over-utilize resources due to replication. Existing modeling-based approaches are hard to rely on for production-level adoption due to modeling errors. We present Wrangler, a system that proactively avoids situations that cause stragglers. Wrangler automatically learns to predict such situations using a statistical learning technique based on cluster resource utilization counters. Furthermore, Wrangler introduces a notion of a confidence measure with these predictions to overcome the modeling error problems; this confidence measure is then exploited to achieve a reliable task scheduling. In particular, by using these predictions to balance delay in task scheduling against the potential for idling of resources, Wrangler achieves a speed up in the overall job completion time. For production-level workloads from Facebook and Clouderas customers, Wrangler improves the 99th percentile job completion time by up to 61% as compared to speculative execution, a widely used straggler mitigation technique. Moreover, Wrangler achieves this speed-up while significantly improving the resource consumption (by up to 55%).

Via: Improving Internet Telephony Call Quality Using Predictive Relay Selection

Interactive real-time streaming applications such as audio-video conferencing, online gaming and app streaming, place stringent requirements on the network in terms of delay, jitter, and packet loss. Many of these applications inherently involve client-to-client communication, which is particularly challenging since the performance requirements need to be met while traversing the public wide-area network (WAN). This is different from the typical situation of cloud-to-client communication, where the WAN can often be bypassed by moving a communication end-point to a cloud “edge”, close to the client. Can we nevertheless take advantage of cloud resources to improve the performance of real-time client-to-client streaming over the WAN? In this paper, we start by analyzing data from a large VoIP provider whose clients are spread across over 21,000 AS’es and nearly all the countries, to understand the challenges faced by interactive audio streaming in the wild. We find that while inter-AS and international paths exhibit significantly worse performance than intra-AS and domestic paths, the pattern of poor performance is nevertheless quite scattered, both temporally and spatially. So any effort to improve performance would have to be fine-grained and dynamic. Then, we turn to the idea of overlay routing, but in the context of the well-provisioned, managed network of a cloud provider rather than peer-to-peer as has been considered in past work. Such a network typically has a global footprint and peers with a large number of network providers. When the performance of a call via the direct path is predicted to be poor, the call traffic could be directed to enter the managed network close to one end point and exit it close to the other end point, thereby avoiding wide-area communication over the public Internet. We present and evaluate data-driven techniques to deciding whether to relay a call through the managed network and if so how to pick the ingress and egress relays to maximize performance, all while operating within a budget for relaying calls via the managed overlay network. We show that call performance can potentially improve by 40%-80% on average, with our techniques closely matching it.

OWeB: A Framework for Offline Web Browsing

Internet browsing is highly dependent on the real-time network availability and speed. This becomes a significant constraint when browsing over slow and intermittent networks. In this paper, we describe a readily deployable system designed for Web browsing over slow, intermittent networks - OWeB, that is minimally dependent on the real-time network availability and requires no changes on the part of the Web servers. The system subscribes to really simple syndication (RSS) feeds from Web servers and pre-fetches all new content as specified in the feed. Since the RSS feeds are published by Web servers they give accurate information about the new and updated content. Efficiency of network usage is achieved by employing standard techniques to handle intermittent networks, and near-complete utilization of all downloaded content results in better resilience in case of interrupted data downloads. We observed a co-relation between the items in an RSS feed and the homepage of the corresponding Web site (i.e.) the feed items essentially define the content section of the homepage. As part of OWeB, we developed an algorithm for automatically extracting the template of home pages and then locally stitching the feed items into the template. This results in Web sites being up-to-date and fully available offline and bandwidth savings, as we only need to download the RSS feed to construct the homepage