Rafal Kocielnik

You Get Who You Pay for: The Impact of Incentives on Participation Bias

Designing effective incentives is a challenge across many social computing contexts, from attracting crowdworkers to sustaining online contributions. However, one aspect of incentivizing that has been understudied is its impact on participation bias, as different incentives may attract different subsets of the population to participate. In this paper, we present two empirical studies in the crowdworking context that show that the incentive offered influence who participates in the task. Using the Basic Human Values, we found that a lottery reward attracted participants who held stronger openness-to-change values while a charity reward attracted those with stronger self-transcendence orientation. Further, we found that participation self-selection resulted in differences in the task outcomes. Through attracting more self-directed individuals, the lottery reward resulted in more ideas generated in a brainstorming task. Design implications include utilizing rewards to target desired participants and using diverse incentives to improve participation diversity.

Calendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop

Although we may complain about meetings, they are an essential part of an information workers work life. Consequently, busy people spend a significant amount of time scheduling meetings. We present Calendar.help, a system that provides fast, efficient scheduling through structured workflows. Users interact with the system via email, delegating their scheduling needs to the system as if it were a human personal assistant. Common scheduling scenarios are broken down using well-defined workflows and completed as a series of microtasks that are automated when possible and executed by a human otherwise. Unusual scenarios fall back to a trained human assistant executing an unstructured macrotask. We describe the iterative approach we used to develop Calendar.help, and share the lessons learned from scheduling thousands of meetings during a year of real-world deployments. Our findings provide insight into how complex information tasks can be broken down into repeatable components that can be executed efficiently to improve productivity.

Send Me a Different Message: Utilizing Cognitive Space to Create Engaging Message Triggers

Social systems and applications often rely on message triggers to promote, remind and even persuade people to perform certain actions. However, repeated exposure to these triggers can lead to boredom, annoyance and decreased engagement. While existing research suggests that diversification of trigger contents may mitigate these issues, no systematic way of introducing it has been proposed. This paper proposes two message diversification strategies based on the use of cognitive spaces: 1) target-diverse -- using concepts cognitively close to the targeted action; and 2) self-diverse -- using concepts cognitively close to the messages recipient. Through a controlled experiment we found that the self-diverse strategy reduces annoyance and boredom from repeated exposure and that both strategies increase perceived informativeness and helpfulness of the triggers. In a subsequent 2-week long field deployment focused on assessing the effects of the self-diverse strategy, we found that this strategy results in higher activity completion through supporting awareness, providing more information, and making the triggers more personally relevant. These diverse triggers are perceived as motivators rather than simple reminders. We conclude with insights on how to design and generate diverse messages.

Aeonium: Visual analytics to support collaborative qualitative coding

Qualitative coding offers the potential to obtain deep insights into social media, but the technique can be inconsistent and hard to scale. Researchers using qualitative coding impose structure on unstructured data through “codes” that represent categories for analysis. Our visual analytics interface, Aeonium, supports human insight in collaborative coding through visual overviews of codes assigned by multiple researchers and distributions of important keywords and codes. The underlying machine learning model highlights ambiguity and inconsistency. Our goal was not to reduce qualitative coding to a machine-solvable problem, but rather to bolster human understanding gained from coding and reinterpreting the data collaboratively. We conducted an experimental study with 39 participants who coded tweets using our interface. In addition to increased understanding of the topic, participants reported that Aeoniums collaborative coding functionality helped them reflect on their own interpretations. Feedback from participants demonstrates that visual analytics can help facilitate rich qualitative analysis and suggests design implications for future exploration.

Designing interactive distance cartograms to support urban travelers

A distance cartogram (DC) is a technique that alters distances between a user-specified origin and the other locations in a map with respect to travel time. With DC, users can weigh the relative travel time costs between the origin and potential destinations at a glance because travel times are projected in a linearly interpolated time space from the origin. Such glance-ability is known to be useful for travelers who are mindful of travel time when finding their travel destinations. When constructing DC, however, uneven urban traffic conditions introduce excessive distortion and challenge user intuition. In addition, there has been little research focusing on DCs user interaction design. To tackle these challenges and realize the potential of DC as an interactive decision-making support tool, we derive a set of useful interactions through two formative studies and devise two novel techniques called Geo-contextual Anchoring Projection and Scalable Road-network Construction. We develop an interactive map system using these techniques and evaluate this system by comparing it against an equidistant map (EM), a widely used conventional layout that preserves the geographical reality. Based on the analysis of user behavior and qualitative feedback, we identify several benefits of using DC itself and of the interaction techniques we derived. We also analyze the specific reasons behind these identified benefits.

Lariat: A Visual Analytics Tool for Social Media Researchers to Explore Twitter Datasets

Online social data is potentially a rich source of insight into human behavior, but the sheer size of these datasets requires specialized tools to facilitate social media research. Visual analytics tools are one promising approach, but calls have been made for more in-depth studies in specific application domains to contribute to the design of such tools. We conducted a formative study to better understand the needs of social media researchers, and created Lariat, a visual analytics tool that facilitates exploratory data analysis through integrated grouping and visualization of social media data. The design of Lariat was informed by the results of the formative study and sensemaking theory, both indicating that the exploratory processes require search, comparison, verification, and iterative refinement. Based on our results and the evaluation of Lariat, we identify a number of design implications for future visual analytics tools in this domain.

Using Machine Learning to Support Qualitative Coding in Social Science: Shifting the Focus to Ambiguity

Machine learning (ML) has become increasingly influential to human society, yet the primary advancements and applications of ML are driven by research in only a few computational disciplines. Even applications that affect or analyze human behaviors and social structures are often developed with limited input from experts outside of computational fields. Social scientists—experts trained to examine and explain the complexity of human behavior and interactions in the world—have considerable expertise to contribute to the development of ML applications for human-generated data, and their analytic practices could benefit from more human-centered ML methods. Although a few researchers have highlighted some gaps between ML and social sciences [51, 57, 70], most discussions only focus on quantitative methods. Yet many social science disciplines rely heavily on qualitative methods to distill patterns that are challenging to discover through quantitative data. One common analysis method for qualitative data is qualitative coding. In this article, we highlight three challenges of applying ML to qualitative coding. Additionally, we utilize our experience of designing a visual analytics tool for collaborative qualitative coding to demonstrate the potential in using ML to support qualitative coding by shifting the focus to identifying ambiguity. We illustrate dimensions of ambiguity and discuss the relationship between disagreement and ambiguity. Finally, we propose three research directions to ground ML applications for social science as part of the progression toward human-centered machine learning.

Challenges and Opportunities for Technology-Supported Activity Reporting in the Workplace

Effective communication of activities and progress in the workplace is crucial for the success of many modern organizations. In this paper, we extend current research on workplace communication and uncover opportunities for technology to support effective work activity reporting. We report on three studies: With a survey of 68 knowledge workers followed by 14 in-depth interviews, we investigated the perceived benefits of different types of progress reports and an array of challenges at three stages: Collection, Composition, and Delivery. We show an important interplay between written and face-to-face reporting, and highlight the importance of tailoring a report to its audience. We then present results from an analysis of 722 reports composed by 361 U.S.-based knowledge workers, looking at the influence of the audience on a reports language. We conclude by discussing opportunities for future technologies to assist both employees and managers in collecting, interpreting, and reporting progress in the workplace.

Beacon: Designing a Portable Device for Self-Administering a Measure of Critical Flicker Frequency

Critical flicker frequency (CFF) is the minimum frequency at which a flickering light source appears fused to an observer. Measuring CFF can support early diagnosis of minimal hepatic encephalopathy (MHE), a condition affecting up to 80% of people with cirrhosis of the liver. However, adoption of CFF measurement in clinical practice has been hampered by the cost of a device for measuring CFF and the need for specialized training to administer the test. This paper presents Beacon, a portable, inexpensive device that enables people to measure their own critical flicker frequency. We adopt a mixed-methods approach to informing and evaluating the design of and potential opportunities for Beacon. We first report on a two-part formative study with 10 participants to evaluate the choice of certain parameters in the design of Beacon. We then report on a study of 41 healthy adults ranging from 18 to 99 years of age, finding that Beacon performs on par with Lafayette Flicker Fusion System, an established medical device, achieving a pearson correlation coefficient of 0.88. We finally report on a focus group with five hepatoligists who work with patients with cirrhosis of the liver, using our initial prototype development to examine their perspectives on potential opportunities and challenges in adoption of a device like Beacon. We discuss Beacon as an exploration of reframing critical flicker frequency measurement from a clinical screening tool into a self-administered self-tracking measure, thereby drawing upon and contributing to research in the health and personal informatics.Critical flicker frequency (CFF) is the minimum frequency at which a flickering light source appears fused to an observer. Measuring CFF can support early diagnosis of minimal hepatic encephalopathy (MHE), a condition affecting up to 80% of people with cirrhosis of the liver. However, adoption of CFF measurement in clinical practice has been hampered by the cost of a device for measuring CFF and the need for specialized training to administer the test. This paper presents Beacon, a portable, inexpensive device that enables people to measure their own critical flicker frequency. We adopt a mixed-methods approach to informing and evaluating the design of and potential opportunities for Beacon. We first report on a two-part formative study with 10 participants to evaluate the choice of certain parameters in the design of Beacon. We then report on a study of 41 healthy adults ranging from 18 to 99 years of age, finding that Beacon performs on par with Lafayette Flicker Fusion System, an established medical device, achieving a pearson correlation coefficient of 0.88. We finally report on a focus group with five hepatoligists who work with patients with cirrhosis of the liver, using our initial prototype development to examine their perspectives on potential opportunities and challenges in adoption of a device like Beacon. We discuss Beacon as an exploration of reframing critical flicker frequency measurement from a clinical screening tool into a self-administered self-tracking measure, thereby drawing upon and contributing to research in the health and personal informatics.

Helping Users Reflect on Their Own Health-Related Behaviors

In this chapter we discuss the use of external sources of data in designing conversational dialogues. We focus on applications in behavior change around physical activity involving dialogues that help users better understand their self-tracking data and motivate healthy behaviors. We start by introducing the areas of behavior change and personal informatics and discussing the importance of self-tracking data in these areas. We then introduce the role of reflective dialogue-based counseling systems in this domain, discuss specific value that self-tracking data can bring, and how it can be used in creating the dialogues. The core of the chapter focuses on six practical examples of design of dialogues involving self-tracking data that we either tested in our research or propose as future directions based on our experiences. We end the chapter by discussing how the design principles for involving external data in conversations can be applied to broader domains. Our goal for this chapter is to share our experiences, outline design principles, highlight several design opportunities in external data-driven computer-based conversations, and encourage the reader to explore creative ways of involving external sources of data in shaping dialogues-based interactions.