Thomas Sandholm

What's inside the Cloud? An architectural map of the Cloud landscape

We propose an integrated Cloud computing stack architecture to serve as a reference point for future mash-ups and comparative studies. We also show how the existing Cloud landscape maps into this architecture and identify an infrastructure gap that we plan to address in future work.

Dynamic proportional share scheduling in Hadoop

We present the Dynamic Priority (DP) parallel task scheduler for Hadoop. It allows users to control their allocated capacity by adjusting their spending over time. This simple mechanism allows the scheduler to make more efficient decisions about which jobs and users to prioritize and gives users the tool to optimize and customize their allocations to fit the importance and requirements of their jobs. Additionally, it gives users the incentive to scale back their jobs when demand is high, since the cost of running on a slot is then also more expensive. We envision our scheduler to be used by deadline or budget optimizing agents on behalf of users. We describe the design and implementation of the DP scheduler and experimental results. We show that our scheduler enforces service levels more accurately and also scales to more users with distinct service levels than existing schedulers.

MapReduce optimization using regulated dynamic prioritization

We present a system for allocating resources in shared data and compute clusters that improves MapReduce job scheduling in three ways. First, the system uses regulated and user-assigned priorities to offer different service levels to jobs and users over time. Second, the system dynamically adjusts resource allocations to fit the requirements of different job stages. Finally, the system automatically detects and eliminates bottlenecks within a job. We show experimentally using real applications that users can optimize not only job execution time but also the cost-benefit ratio or prioritization efficiency of a job using these three strategies. Our approach relies on a proportional share mechanism that continuously allocates virtual machine resources. Our experimental results show a 11-31% improvement in completion time and 4-187% improvement in prioritization efficiency for different classes of MapReduce jobs. We further show that delay intolerant users gain even more from our system.

Collective attention and the dynamics of group deals

We present a study of the group purchasing behavior of daily deals in Groupon and LivingSocial and formulate a predictive dynamic model of collective attention for group buying behavior. Using large data sets from both Groupon and LivingSocial we show how the model is able to predict the success of group deals as a function of time.We find that Groupon deals are easier to predict accurately earlier in the deal lifecycle than LivingSocial deals due to the total number of deal purchases saturating quicker. One possible explanation for this is that the incentive to socially propagate a deal is based on an individual threshold in LivingSocial, whereas in Groupon it is based on a collective threshold which is reached very early. Furthermore, the personal benefit of propagating a deal is greater in LivingSocial.

Real-time, location-aware collaborative filtering of web content

In this paper we describe the collaborative filtering feature of a location-aware, Web content recommendation service, called Gloe. The main purpose of our collaborative filtering solution is to increase the diversity of recommendations and to thereby mitigate popularity bias. The key challenge is to filter candidate suggestions in real-time, with minimal data mining and model building overhead. There is an apparent trade-off between building general purpose reusable models with contributions from a large user base on one hand and efficient on-line evaluation and recommendation in realtime on the other hand. Our solution is to apply item-based, top-N collaborative filtering within a hierarchical folksonomy structure in a Geohash pre-partitioned geographic locale. We demonstrate that these recommendations can be, on average, as fast to compute as aggregate rating-based recommendations, while offering a more diverse as well as personalized set of recommendations.

Notes on Cloud computing principles

This letter provides a review of fundamental distributed systems and economic Cloud computing principles. These principles are frequently deployed in their respective fields, but their interdependencies are often neglected. Given that Cloud Computing first and foremost is a new business model, a new model to sell computational resources, the understanding of these concepts is facilitated by treating them in unison. Here, we review some of the most important concepts and how they relate to each other.

Foxtrot: a soundtrack for where you are

In this paper, we present a mobile location-aware and crowd-sourced audio application, Foxtrot, that allows people to share the sounds and music they enjoy and associate with a particular location. Foxtrot plays an automatically created radio-like stream of geo-tagged music, ambient sounds and comments left by friends and other people. We discuss some of the design considerations for Foxtrot and our approach to selecting and scheduling audio content for playback. In addition, we present a pilot study of Foxtrot, which indicates that a location-aware music system might indeed provide an engaging mobile experience for users.

SocRoutes: safe routes based on tweet sentiments

Location-based services, and in particular personal navigation systems, have become increasingly popular with the widespread use of GPS technology in smart devices. Existing navigation systems are designed to suggest routes based on the shortest distance or the fastest time to a target. In this paper, we propose a new type of route navigation based on regional context---primarily sentiments. Our system, called SocRoutes, aims to find a safer, friendlier, and more enjoyable route based on sentiments inferred from real-time, geotagged messages from Twitter. SocRoutes tailors routes by avoiding places with extremely negative sentiments, thereby potentially finding a safer and more enjoyable route with marginal increase in total distance compared to the shortest path. The system supports three types of traveling modes: walking, bicycling, and driving. We validated the idea based on crime history data from the City of Chicago Portal in December 2012, and sentiments extracted from geotagged tweets during the same time. We discovered that there was a significant correlation between regional Twitter posting sentiments and crime rate, in particular for high-crime and highly negative sentiment areas. We also demonstrated that SocRoutes, by solely utilizing social media sentiments, can find routes that bypass crime hotspots.

Global budgets for local recommendations

We present the design, implementation and evaluation of a new geotagging service, Gloe, that makes it easy to find, rate and recommend arbitrary on-line content in a mobile setting. The service automates the content search process by taking advantage of geographic and social context, while using crowdsourced expertise to present a personalized feed of targeted information ranked by a novel geo-aware rating and incentive mechanism. Users rate the relevance of recommendations for particular locations using a limited, global voting budget. This budget is, in turn, increased by accurately predicting local content popularity. One of the key goals of our mechanism is to encourage ratings, and in an evaluation of the live system we found that the rating to click ratio was 107 times higher than the ratio for videos on YouTube, 34 times higher than the ratio for applications on the Android Market, and 3 times higher than the ratio for Web pages on Digg. To investigate whether our mechanism also had qualitative effects on the ratings we conducted a number of experiments on Amazon Mechanical Turk, with 500 users, comparing our mechanism to the de-facto 5-star ratings commonly in use on the Web. Our results show that budgets improved the ranking and incentives improved the aggregate rating of a series of location-dependent Web pages.

An On-Demand WebRTC and IoT Device Tunneling Service for Hospitals

In this paper we present the implementation of a WebRTC gateway service that can forward ad-hoc RTP data plane traffic from a browser inside a local hospital network to a browser on a local home network. The gateway leverages the same infrastructure used by the hospital to tunnel sensor and control data for medical devices in home-care deployments. In our use case, doctors at hospitals can only access port 80 through the hospital firewall on external machines, and they need to communicate with patients who are typically behind a NAT in a local WiFi network. VPN solutions only work for staff but not between patients and staff. Our solution solves this problem by redirecting all WebRTC traffic through a gateway service on the local network that has a secure tunnel established with a public gateway. The public gateway redirects traffic from multiple concurrent streams securely between local gateway services that connect to it. The local gateways also communicate with browsers on their local network to mimic a direct browser-to-browser connection without having to change the browser runtime. We have demonstrated that this technique works well within the hospital network and arbitrary patient networks, without the need for any individual host configuration. In our evaluation we show that the latency overhead is 18-20 ms for each concurrent stream added to the same gateway service, which is not discernible with a naked eye until you have more than 10 concurrent streams.