Anhong Guo

WearWrite: Crowd-Assisted Writing from Smartwatches

The physical constraints of smartwatches limit the range and complexity of tasks that can be completed. Despite interface improvements on smartwatches, the promise of enabling productive work remains largely unrealized. This paper presents WearWrite, a system that enables users to write documents from their smartwatches by leveraging a crowd to help translate their ideas into text. WearWrite users dictate tasks, respond to questions, and receive notifications of major edits on their watch. Using a dynamic task queue, the crowd receives tasks issued by the watch user and generic tasks from the system. In a week-long study with seven smartwatch users supported by approximately 29 crowd workers each, we validate that it is possible to manage the crowd writing process from a watch. Watch users captured new ideas as they came to mind and managed a crowd during spare moments while going about their daily routine. WearWrite represents a new approach to getting work done from wearables using the crowd.

VizLens: A Robust and Interactive Screen Reader for Interfaces in the Real World

The world is full of physical interfaces that are inaccessible to blind people, from microwaves and information kiosks to thermostats and checkout terminals. Blind people cannot independently use such devices without at least first learning their layout, and usually only after labeling them with sighted assistance. We introduce VizLens - an accessible mobile application and supporting backend that can robustly and interactively help blind people use nearly any interface they encounter. VizLens users capture a photo of an inaccessible interface and send it to multiple crowd workers, who work in parallel to quickly label and describe elements of the interface to make subsequent computer vision easier. The VizLens application helps users recapture the interface in the field of the camera, and uses computer vision to interactively describe the part of the interface beneath their finger (updating 8 times per second). We show that VizLens provides accurate and usable real-time feedback in a study with 10 blind participants, and our crowdsourcing labeling workflow was fast (8 minutes), accurate (99.7%), and cheap (

Facade: Auto-generating Tactile Interfaces to Appliances

1.15). We then explore extensions of VizLens that allow it to (i) adapt to state changes in dynamic interfaces, (ii) combine crowd labeling with OCR technology to handle dynamic displays, and (iii) benefit from head-mounted cameras. VizLens robustly solves a long-standing challenge in accessibility by deeply integrating crowdsourcing and computer vision, and foreshadows a future of increasingly powerful interactive applications that would be currently impossible with either alone.

CapAuth: Identifying and Differentiating User Handprints on Commodity Capacitive Touchscreens

Digital keypads have proliferated on common appliances, from microwaves and refrigerators to printers and remote controls. For blind people, such interfaces are inaccessible. We conducted a formative study with 6 blind people which demonstrated a need for custom designs for tactile labels without dependence on sighted assistance. To address this need, we introduce Facade - a crowdsourced fabrication pipeline to make physical interfaces accessible by adding a 3D printed augmentation of tactile buttons overlaying the original panel. Blind users capture a photo of an inaccessible interface with a standard marker for absolute measurements using perspective transformation. Then this image is sent to multiple crowd workers, who work in parallel to quickly label and describe elements of the interface. These labels are then used to generate 3D models for a layer of tactile and pressable buttons that fits over the original controls. Users can customize the shape and labels of the buttons using a web interface. Finally, a consumer-grade 3D printer fabricates the layer, which is then attached to the interface using adhesives. Such fabricated overlay is an inexpensive (

Order Picking with Head-Up Displays

10) and more general solution to making physical interfaces accessible.

WearWrite: Orchestrating the Crowd to Complete Complex Tasks from Wearables

User identification and differentiation have implications in many application domains, including security, personalization, and co-located multiuser systems. In response, dozens of approaches have been developed, from fingerprint and retinal scans, to hand gestures and RFID tags. In this work, we propose CapAuth, a technique that uses existing, low-level touchscreen data, combined with machine learning classifiers, to provide real-time authentication and even identification of users. As a proof-of-concept, we ran our software on an off-the-shelf Nexus 5 smartphone. Our user study demonstrates twenty-participant authentication accuracies of 99.6%. For twenty-user identification, our software achieved 94.0% accuracy and 98.2% on groups of four, simulating family use.

VizMap: Accessible Visual Information Through Crowdsourced Map Reconstruction

Experiments suggest that using head-up displays like Google Glass to support parts picking for distribution results in fewer errors than current processes. Making Glass opaque instead of transparent further improves selection efficiency. The Web extra at http://youtu.be/yUZFaCP6rP4 is a video demonstrating that order picking assisted by head-up display (HUD) is faster, has fewer errors, requires less workload, and is preferred to order picking assisted by pick-by-light, paper pick list, and cart-mounted display (CMD). The second Web extra at http://youtu.be/RApBJ0U3XpI is a video demonstrating that order picking assisted by a Google Glass with an opaque display is three percent faster than with a transparent display (default).

Comparing order picking assisted by head-up display versus pick-by-light with explicit pick confirmation

Smartwatches are becoming increasingly powerful, but limited input makes completing complex tasks impractical. Our WearWrite system introduces a new paradigm for enabling a watch user to contribute to complex tasks, not through new hardware or input methods, but by directing a crowd to work on their behalf from their wearable device. WearWrite lets authors give writing instructions and provide bits of expertise and big picture directions from their smartwatch, while crowd workers actually write the document on more powerful devices. We used this approach to write three academic papers, and found it was effective at producing reasonable drafts.

A comparison of order picking methods augmented with weight checking error detection

When navigating indoors, blind people are often unaware of key visual information, such as posters, signs, and exit doors. Our VizMap system uses computer vision and crowdsourcing to collect this information and make it available non-visually. VizMap starts with videos taken by on-site sighted volunteers and uses these to create a 3D spatial model. These video frames are semantically labeled by remote crowd workers with key visual information. These semantic labels are located within and embedded into the reconstructed 3D model, forming a query-able spatial representation of the environment. VizMap can then localize the user with a photo from their smartphone, and enable them to explore the visual elements that are nearby. We explore a range of example applications enabled by our reconstructed spatial representation. With VizMap, we move towards integrating the strengths of the end user, on-site crowd, online crowd, and computer vision to solve a long-standing challenge in indoor blind exploration.

Beyond the Touchscreen: An Exploration of Extending Interactions on Commodity Smartphones

Manual order picking is an important part of distribution. Many techniques have been proposed to improve pick efficiency and accuracy. Previous studies compared pick-by-HUD (Head-Up Display) with pick-by-light but without the explicit pick confirmation that is typical in industrial environments. We compare a pick-by-light system designed to emulate deployed systems with a pick-by-HUD system using Google Glass. The pick-by-light system tested 50% slower than pick-by-HUD and required a higher workload. The number of errors committed and picker preference showed no statistically significant difference.