Max Chickering

ModelTracker: Redesigning Performance Analysis Tools for Machine Learning

Model building in machine learning is an iterative process. The performance analysis and debugging step typically involves a disruptive cognitive switch from model building to error analysis, discouraging an informed approach to model building. We present ModelTracker, an interactive visualization that subsumes information contained in numerous traditional summary statistics and graphs while displaying example-level performance and enabling direct error examination and debugging. Usage analysis from machine learning practitioners building real models with ModelTracker over six months shows ModelTracker is used often and throughout model building. A controlled experiment focusing on ModelTrackers debugging capabilities shows participants prefer ModelTracker over traditional tools without a loss in model performance.

Mining recommendations from the web

In this paper we study the challenges and evaluate the effectiveness of data collected from the web for recommendations. We provide experimental results, including a user study, showing that our methods produce good recommendations in realistic applications. We propose a new evaluation metric, that takes into account the difficulty of prediction. We show that the new metric aligns well with the results from a user study.

Fast algorithms for finding matchings in lopsided bipartite graphs with applications to display ads

We derive efficient algorithms for both detecting and representing matchings in lopsided bipartite graphs; such graphs have so many nodes on one side that it is infeasible to represent them in memory or to identify matchings using standard approaches. Detecting and representing matchings in lopsided bipartite graphs is important for allocating and delivering guaranteed-placement display ads, where the corresponding bipartite graph of interest has nodes representing advertisers on one side and nodes representing web-page impressions on the other; real-world instances of such graphs can have billions of impression nodes. We provide theoretical guarantees for our algorithms, and in a real-world advertising application, we demonstrate the feasibility of our detection algorithms.

Market design & analysis for a P2P backup system

In this paper we take the problem of a market-based P2P backup application and carry it through market design, to implementation, to theoretical and experimental analysis. While the long-term goal is an open market using real money, here we consider a system where monetary transfers are prohibited. We first describe the design of the P2P resource exchange market and the UI we developed. Second, we prove theorems on equilibrium existence and uniqueness. Third, we prove a surprising impossibility result regarding the limited controllability of the equilibrium and show how to address this. Fourth, we present a price update algorithm that uses daily supply and demand information to move prices towards the equilibrium and we provide a theoretical and experimental convergence analysis. The market design described in this paper is already implemented as part of a Microsoft research project on P2P backup systems and an alpha version of the software has been successfully tested.

Budget smoothing for internet ad auctions: a game theoretic approach

In Internet ad auctions, search engines often throttle budget constrained advertisers so as to spread their spends across the specified time period. Such policies are known as budget smoothing policies. In this paper, we perform a principled, game-theoretic study of what the outcome of an ideal budget smoothing algorithm should be. In particular, we propose the notion of regret-free budget smoothing policies whose outcomes throttle each advertiser optimally, given the participation of the other advertisers. We show that regret-free budget smoothing policies always exist, and in the case of single slot auctions we can give a polynomial time smoothing algorithm. Inspired by the existence proof, we design a heuristic for budget smoothing which performs considerably better than existing benchmark heuristics.

Intentions: a game for classifying search query intent

Knowing the intent of a search query allows for more intelligent ways of retrieving relevant search results. Most of the recent work on automatic detection of query intent uses supervised learning methods that require a substantial amount of labeled data; manually collecting such data is often time-consuming and costly. Human computation is an active research area that includes studies of how to build online games that people enjoy playing, while in the process providing the system with useful data. In this work, we present the design principles behind a new game called Intentions, which aims to collect data about the intent behind search queries.

Handling Out-of-Grammar Commands in Mobile Speech Interaction Using Backoff Filler Models

In command and control (C&C) speech interaction, users interact by speaking commands or asking questions typically specified in a context-free grammar (CFG). Unfortunately, users often produce out-of-grammar (OOG) commands, which can result in misunderstanding or nonunderstanding. We explore a simple approach to handling OOG commands that involves generating a backoff grammar from any CFG using filler models, and utilizing that grammar for recognition whenever the CFG fails. Working within the memory footprint requirements of a mobile C&C product, applying the approach yielded a 35% relative reduction in semantic error rate for OOG commands. It also improved partial recognitions for enabling clarification dialogue.

Market Design for a P2P Backup System

Peer-to-peer (P2P) backup systems are an attractive alternative to server-based systems because the immense costs of large data centers can be saved by using idle resources on millions of private computers instead. This paper presents the design and theoretical analysis of a market for a P2P backup system. While our long-term goal is an open resource exchange market using real money, here we consider a system where monetary transfers are prohibited. A user who wants to backup his data must in return supply some of his resources (storage space, upload and download bandwidth) to the system.We propose a hybrid P2P architecture where all backup data is transferred directly between peers, but a dedicated server coordinates all operations and maintains meta-data. We achieve high reliability guarantees while keeping our data replication factor low by adopting sophisticated erasure coding technology (cf., [2]).

A Bayesian Approach to Tackling Hard Computational Problems (Preliminary Report)

Abstract Abstract We describe research and results centering on the construction and use of Bayesian models that can predict the run time of problem solvers. Our efforts are motivated by observations of high variance in the run time uired to solve instances for several challenging problems. The methods have application to the decision-theoretic control of hard search and reasoning algorithms. We illustrate the approach with a focus on the task of predicting run time for general and domain-specific solvers on a hard class of structured constraint satisfaction problems. We describe the use of learned models to predict the ultimate length of a trial, based on observing the behavior of the search algorithm during an early phase of a problem session. Finally, we discuss how we can employ the models to inform dynamic run-time decisions. We thank Dimitris Achlioptas for his insightful contributions and feedback.

Rapidly Deploying Grammar-Based Speech Applications with Active Learning and Back-off Grammars

Grammar-based approaches to spoken language understanding are utilized to a great extent in industry, particularly when developers are confronted with data sparsity. In order to ensure wide grammar coverage, developers typically modify their grammars in an iterative process of deploying the application, collecting and transcribing user utterances, and adjusting the grammar. In this paper, we explore enhancing this iterative process by leveraging active learning with back-off grammars. Because the back-off grammars expand coverage of user utterances, developers have a safety net for deploying applications earlier. Furthermore, the statistics related to the back-off can be used for active learning, thus reducing the effort and cost of data transcription. In experiments conducted on a commercially deployed application, the approach achieved levels of semantic accuracy comparable to transcribing all failed utterances with 87% less transcriptions.