Saleema Amershi

Regroup: interactive machine learning for on-demand group creation in social networks

We present ReGroup, a novel end-user interactive machine learning system for helping people create custom, on demand groups in online social networks. As a person adds members to a group, ReGroup iteratively learns a probabilistic model of group membership specific to that group. ReGroup then uses its currently learned model to suggest additional members and group characteristics for filtering. Our evaluation shows that ReGroup is effective for helping people create large and varied groups, whereas traditional methods (searching by name or selecting from an alphabetical list) are better suited for small groups whose members can be easily recalled by name. By facilitating on demand group creation, ReGroup can enable in-context sharing and potentially encourage better online privacy practices. In addition, applying interactive machine learning to social network group creation introduces several challenges for designing effective end-user interaction with machine learning. We identify these challenges and discuss how we address them in ReGroup.

Amplifying community content creation with mixed initiative information extraction

Although existing work has explored both information extraction and community content creation, most research has focused on them in isolation. In contrast, we see the greatest leverage in the synergistic pairing of these methods as two interlocking feedback cycles. This paper explores the potential synergy promised if these cycles can be made to accelerate each other by exploiting the same edits to advance both community content creation and learning-based information extraction. We examine our proposed synergy in the context of Wikipedia infoboxes and the Kylin information extraction system. After developing and refining a set of interfaces to present the verification of Kylin extractions as a non primary task in the context of Wikipedia articles, we develop an innovative use of Web search advertising services to study people engaged in some other primary task. We demonstrate our proposed synergy by analyzing our deployment from two complementary perspectives: (1) we show we accelerate community content creation by using Kylins information extraction to significantly increase the likelihood that a person visiting a Wikipedia article as a part of some other primary task will spontaneously choose to help improve the articles infobox, and (2) we show we accelerate information extraction by using contributions collected from people interacting with our designs to significantly improve Kylins extraction performance.

Automatic recognition of learner groups in exploratory learning environments

In this paper, we present the application of unsupervised learning techniques to automatically recognize behaviors that may be detrimental to learning during interaction with an Exploratory Learning Environment (ELE). First, we describe how we use the k-means clustering algorithm for off-line identification of learner groups with distinguishing interaction patterns who also show similar learning improvements with an ELE. We then discuss how a k-means on-line classifier, trained with the learner groups detected off-line, can be used for adaptive support in ELEs. We aim to show the value of a data-based approach for recognizing learners as an alternative to knowledge-based approaches that tend to be complex and time-consuming even for domain experts, especially in highly unstructured ELEs.

ModelTracker: Redesigning Performance Analysis Tools for Machine Learning

Model building in machine learning is an iterative process. The performance analysis and debugging step typically involves a disruptive cognitive switch from model building to error analysis, discouraging an informed approach to model building. We present ModelTracker, an interactive visualization that subsumes information contained in numerous traditional summary statistics and graphs while displaying example-level performance and enabling direct error examination and debugging. Usage analysis from machine learning practitioners building real models with ModelTracker over six months shows ModelTracker is used often and throughout model building. A controlled experiment focusing on ModelTrackers debugging capabilities shows participants prefer ModelTracker over traditional tools without a loss in model performance.

CueT: human-guided fast and accurate network alarm triage

Network alarm triage refers to grouping and prioritizing a stream of low-level device health information to help operators find and fix problems. Today, this process tends to be largely manual because existing tools cannot easily evolve with the network. We present CueT, a system that uses interactive machine learning to learn from the triaging decisions of operators. It then uses that learning in novel visualizations to help them quickly and accurately triage alarms. Unlike prior interactive machine learning systems, CueT handles a highly dynamic environment where the groups of interest are not known a-priori and evolve constantly. A user study with real operators and data from a large network shows that CueT significantly improves the speed and accuracy of alarm triage compared to the networks current practice.

Structured labeling for facilitating concept evolution in machine learning

Labeling data is a seemingly simple task required for training many machine learning systems, but is actually fraught with problems. This paper introduces the notion of concept evolution, the changing nature of a persons underlying concept (the abstract notion of the target class a person is labeling for, e.g., spam email, travel related web pages) which can result in inconsistent labels and thus be detrimental to machine learning. We introduce two structured labeling solutions, a novel technique we propose for helping people define and refine their concept in a consistent manner as they label. Through a series of five experiments, including a controlled lab study, we illustrate the impact and dynamics of concept evolution in practice and show that structured labeling helps people label more consistently in the presence of concept evolution than traditional labeling.

Unsupervised and supervised machine learning in user modeling for intelligent learning environments

In this research, we outline a user modeling framework that uses both unsupervised and supervised machine learning in order to reduce development costs of building user models, and facilitate transferability. We apply the framework to model student learning during interaction with the Adaptive Coach for Exploration (ACE) learning environment (using both interface and eye-tracking data). In addition to demonstrating framework effectiveness, we also compare results from previous research on applying the framework to a different learning environment and data type. Our results also confirm previous research on the value of using eye-tracking data to assess student learning.

Co-located collaborative web search: understanding status quo practices

Co-located collaborative Web search is a surprisingly common activity, despite the fact that Web browsers and search engines are not designed to support collaboration. We report the findings of two studies (a diary study and an observational study) that provide insights regarding the frequency of co-located collaborative searching, the strategies participants use, and the pros and cons of these strategies. We then articulate design implications for next-generation tools that could enhance the experience of co-located collaborative search.

Designing for effective end-user interaction with machine learning

End-user interactive machine learning is a promising tool for enhancing human capabilities with large data. Recent work has shown that we can create end-user interactive machine learning systems for specific applications. However, we still lack a generalized understanding of how to design effective end-user interaction with interactive machine learning systems. My dissertation work aims to advance our understanding of this question by investigating new techniques that move beyond naïve or ad-hoc approaches and balance the needs of both end-users and machine learning algorithms. Although these explorations are grounded in specific applications, we endeavored to design strategies independent of application or domain specific features. As a result, our findings can inform future end-user interaction with machine learning systems.

Comparing web interaction models in developing regions

Internet connections in developing regions are scarce and often unreliable. While options for connecting to the Internet are gradually being realized, progress is slow. We observed people performing web search and browsing in a low bandwidth environment in Kerala, India. We found that people in this environment experienced frustration and boredom while waiting for page loads compared to typical experiences in the developed world. Following these observations, we conducted a formal study with 20 participants at the same location comparing the conventional web search and browsing process with an asynchronous queueing model. Participants using the asynchronous queueing system performed as well as the status quo in terms of the number of tasks completed, and we observed greater interaction and information viewed for the asynchronous system. Our participants also preferred the asynchronous system over conventional search. Finally, we found evidence that the asynchronous system would have greater benefits in environments where the network is even more constrained.