Edward Lank

User-defined motion gestures for mobile interaction

Modern smartphones contain sophisticated sensors to monitor three-dimensional movement of the device. These sensors permit devices to recognize motion gestures - deliberate movements of the device by end-users to invoke commands. However, little is known about best-practices in motion gesture design for the mobile computing paradigm. To address this issue, we present the results of a guessability study that elicits end-user motion gestures to invoke commands on a smartphone device. We demonstrate that consensus exists among our participants on parameters of movement and on mappings of motion gestures onto commands. We use this consensus to develop a taxonomy for motion gestures and to specify an end-user inspired motion gesture set. We highlight the implications of this work to the design of smartphone applications and hardware. Finally, we argue that our results influence best practices in design for all gestural interfaces.

Endpoint prediction using motion kinematics

Recently proposed novel interaction techniques such as cursor jumping [1] and target expansion for tiled arrangements [13] are predicated on an ability to effectively estimate the endpoint of an input gesture prior to its completion. However, current endpoint estimation techniques lack the precision to make these interaction techniques possible. To address a recognized lack of effective endpoint prediction mechanisms, we propose a new technique for endpoint prediction that applies established laws of motion kinematics in a novel way to the identification of motion endpoint. The technique derives a model of speed over distance that permits extrapolation. We verify our model experimentally using stylus targeting tasks, and demonstrate that our endpoint prediction is almost twice as accurate as the previously tested technique [13] at points more than twice as distant from motion endpoint.

Renegade gaming: practices surrounding social use of the Nintendo DS handheld gaming system

Todays handheld gaming systems allow players to engage in multiplayer games via ad-hoc, wireless networking. They are also now sufficiently commonplace that it is possible to study how portability and ad-hoc wireless networking have affected the social gaming practices of owners of these systems. In this paper, we report findings from a qualitative study investigating the collocated multiplayer gaming practices of Nintendo DS owners. Based on interviews of nine DS owners and observations of three organized gaming events, we identified three major themes surrounding the social, multiplayer gaming practices of Nintendo DS users: renegade gaming, or the notion that users reappropriate contexts traditionally hostile to game play; pragmatic and social barriers to the formation of ad-hoc pick-up games, despite a clear desire for multiplayer, collocated gaming; and private gaming spheres, or the observation that the handheld devices form factor creates individual, privatized gaming contexts within larger social contexts. These findings lead to a set of implications for the design of future handheld gaming systems.

MathBrush: A System for Doing Math on Pen-Based Devices

Many on-line (interactive) mathematics recognition systems allow the creation of typeset equations, normally in LaTeX, but they do not support mathematical problem solving. In this paper, we present MathBrush, a system that allows users to draw math input using a pen-input device on a tablet computer, recognizes the math expression, and then supports mathematical transformation and problem solving using back-end Computer Algebra Systems (CAS). We describe the architecture of the MathBrush system, which includes modules that support symbol recognition, semantic analysis, the transfer of recognized expressions to back-end CAS, and interface techniques for interacting with CAS output. We also identify unique challenges associated with recognition for math problem solving, such as the need for deeper semantic analysis than is required by LATEX, and the need to deal with ambiguities in user input. Our experiences serve to inform researchers seeking to design interactive mathematics recognition systems geared toward mathematical problem solving.

Tap, swipe, or move: attentional demands for distracted smartphone input

Smartphones are frequently used in environments where the user is distracted by another task, for example by walking or by driving. While the typical interface for smartphones involves hardware and software buttons and surface gestures, researchers have recently posited that, for distracted environments, benefits may exist in using motion gestures to execute commands. In this paper, we examine the relative cognitive demands of motion gestures and surface taps and gestures in two specific distracted scenarios: a walking scenario, and an eyes-free seated scenario. We show, first, that there is no significant difference in reaction time for motion gestures, taps, or surface gestures on smartphones. We further show that motion gestures result in significantly less time looking at the smartphone during walking than does tapping on the screen, even with interfaces optimized for eyes-free input. Taken together, these results show that, despite somewhat lower throughput, there may be benefits to making use of motion gestures as a modality for distracted input on smartphones.

Grammar-based techniques for creating ground-truthed sketch corpora

Although publicly available, ground-truthed corpora have proven useful for training, evaluating, and comparing recognition systems in many domains, the availability of such corpora for sketch recognizers, and math recognizers in particular, is currently quite poor. This paper presents a general approach to creating large, ground-truthed corpora for structured sketch domains such as mathematics. In the approach, random sketch templates are generated automatically using a grammar model of the sketch domain. These templates are transcribed manually, then automatically annotated with ground-truth. The annotation procedure uses the generated sketch templates to find a matching between transcribed and generated symbols. A large, ground-truthed corpus of handwritten mathematical expressions presented in the paper illustrates the utility of the approach.

Focus+Context sketching on a pocket PC

Current Personal Digital Assistants (PDAs) such as Pocket PCs provide little support for sketching. This is due, no doubt, to the limited screen size and relatively poor resolution of these devices. To support sketching on these devices, we describe a Focus+Context technique using fisheye views integrated into a sketching application on Pocket PCs. Our technique creates a distorted display view of a canvas that is larger than a Pocket PC screen. Users draw in a high resolution focus area that is smoothly embedded into the distorted context. We have conducted a modest user trial that validates user preference for this drawing technique over a traditional scroll-bar based interface.

Sloppy selection: Providing an accurate interpretation of imprecise selection gestures

This paper describes on-going work in the analysis of motion dynamics in pen-based interaction. The overall goal is the creation of a model of user motion in pen gestures where constraint and curvature vary over the length of the path. In particular, speed/curvature models of motion are used to analyze pen trajectories and infer target constraints obeyed by a user performing selection gestures. We aim to use this information to calculate an effective local spatial selection tolerance associated with each gesture. This can be used to perform selection according to user intent instead of their literal stroke. Here, we describe our early analysis of constrained user selection gestures, and outline a prototype application that infers a tolerance for one type of selection gesture. The application selectively splits pen strokes based on an analysis of user motion.

A retargetable framework for interactive diagram recognition

The design of new diagram recognition systems remains a challenging problem. Ideally, recognition systems should accept real-world input, perform robustly, fail gracefully, and be implemented in a timely manner. In reality, the intricacy involved in implementing recognition systems for diagram notations makes this a challenging open problem. One solution to these challenges is the design of middleware to speed the development of robust applications. Middleware takes the form of a framework or toolkit for the creation of applications. This paper describes a retargetable framework which can be used to speed the development of robust interactive sketch recognition systems. The system includes a drawing surface to capture interactively created drawings, a set of generic segmentation routines, a character recognizer, and a common interface for integrating domain-specific components. The framework has been used to construct systems for the recognition of UML, math, and molecular diagrams. Work is on-going on the design of additional generic recognizers of logical structure and spatial layout of diagrams.

Territoriality and behaviour on and around large vertical publicly-shared displays

We investigate behaviours on, and around, large vertical displays during concurrent usage. Using an observational field study, we identify fundamental patterns of how people use existing public displays: their orientation, positioning, group identification, and behaviour within and between social groups just-before, during, and just-after usage. These results are then used to motivate a controlled experiment where two individuals or two pairs of individuals complete tasks concurrently on a simulated large vertical display. Results from our controlled study demonstrates that vertical surface territories are similar to those found in horizontal tabletops in function, but their definitions and social conventions are different. In addition, the nature of use-while-standing systems results in more complex and dynamic physical territories around the display. We show that the anthropological notion of personal space must be slightly refined for application to vertical displays.