Gilles Bailly

iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing

We propose iSkin, a novel class of skin-worn sensors for touch input on the body. iSkin is a very thin sensor overlay, made of biocompatible materials, and is flexible and stretchable. It can be produced in different shapes and sizes to suit various locations of the body such as the finger, forearm, or ear. Integrating capacitive and resistive touch sensing, the sensor is capable of detecting touch input with two levels of pressure, even when stretched by 30% or when bent with a radius of 0.5cm. Furthermore, iSkin supports single or multiple touch areas of custom shape and arrangement, as well as more complex widgets, such as sliders and click wheels. Recognizing the social importance of skin, we show visual design patterns to customize functional touch sensors and allow for a visually aesthetic appearance. Taken together, these contributions enable new types of on-body devices. This includes finger-worn devices, extensions to conventional wearable devices, and touch input stickers, all fostering direct, quick, and discreet input for mobile computing.

ShoeSense: a new perspective on gestural interaction and wearable applications

When the user is engaged with a real-world task it can be inappropriate or difficult to use a smartphone. To address this concern, we developed ShoeSense, a wearable system consisting in part of a shoe-mounted depth sensor pointing upward at the wearer. ShoeSense recognizes relaxed and discreet as well as large and demonstrative hand gestures. In particular, we designed three gesture sets (Triangle, Radial, and Finger-Count) for this setup, which can be performed without visual attention. The advantages of ShoeSense are illustrated in five scenarios: (1) quickly performing frequent operations without reaching for the phone, (2) discreetly performing operations without disturbing others, (3) enhancing operations on mobile devices, (4) supporting accessibility, and (5) artistic performances. We present a proof-of-concept, wearable implementation based on a depth camera and report on a lab study comparing social acceptability, physical and mental demand, and user preference. A second study demonstrates a 94-99% recognition rate of our recognizers.

Flower menus: a new type of marking menu with large menu breadth, within groups and efficient expert mode memorization

This paper presents Flower menu, a new type of Marking menu that does not only support straight, but also curved gestures for any of the 8 usual orientations. Flower menus make it possible to put many commands at each menu level and thus to create as large a hierarchy as needed for common applications. Indeed our informal analysis of menu breadth in popular applications shows that a quarter of them have more than 16 items. Flower menus can easily contain 20 items and even more (theoretical maximum of 56 items). Flower menus also support within groups as well as hierarchical groups. They can thus favor breadth organization (within groups) or depth organization (hierarchical groups): as a result, the designers can lay out items in a very flexible way in order to reveal meaningful item groupings. We also investigate the learning performance of the expert mode of Flower menus. A user experiment is presented that compares linear menus (baseline condition), Flower menus and Polygon menus, a variant of Marking menus that supports a breadth of 16 items. Our experiment shows that Flower menus are more efficient than both Polygon and Linear menus for memorizing command activation in expert mode.

Finger-count & radial-stroke shortcuts: 2 techniques for augmenting linear menus on multi-touch surfaces

We propose Radial-Stroke and Finger-Count Shortcuts, two techniques aimed at augmenting the menubar on multi-touch surfaces. We designed these multi-finger two-handed interaction techniques in an attempt to overcome the limitations of direct pointing on interactive surfaces, while maintaining compatibility with traditional interaction techniques. While Radial-Stroke Shortcuts exploit the well-known advantages of Radial Strokes, Finger-Count Shortcuts exploit multi-touch by simply counting the number of fingers of each hand in contact with the surface. We report the results of an experimental evaluation of our technique, focusing on expert-mode performance. Finger-Count Shortcuts outperformed Radial-Stroke Shortcuts in terms of both easiness of learning and performance speed.

Cuenesics: using mid-air gestures to select items on interactive public displays

Most of todays public displays only show predefined contents that passers-by are not able to change. We argue that interactive public displays would benefit from immediately usable mid-air techniques for choosing options, expressing opinions or more generally selecting one among several items. We propose a design space for hand-gesture based mid-air selection techniques on interactive public displays, along with four specific techniques that we evaluated at three different locations in the the field. Our findings include: 1) if no hint is provided, people successfully use Point+Dwell for selecting items, 2) the user representation could be switched from Mirror to Cursor after registration without causing confusion, 3) people tend to explore items before confirming one, 4) in a public context, people frequently interact inadvertently (without looking at the screen). We conclude by providing recommendations for designers of interactive public displays to support immediate usability for mid-air selection.

Comparing free hand menu techniques for distant displays using linear, marking and finger-count menus

Distant displays such as interactive public displays (IPD) or interactive television (ITV) require new interaction techniques as traditional input devices may be limited or missing in these contexts. Free hand interaction, as sensed with computer vision techniques, presents a promising interaction technique. This paper presents the adaptation of three menu techniques for free hand interaction: Linear menu, Marking menu and Finger-Count menu. The first study based on a Wizard-of-Oz protocol focuses on Finger-Counting postures in front of interactive television and public displays. It reveals that participants do not choose the most efficient gestures neither before nor after the experiment. Results are used to develop a Finger-Count recognizer. The second experiment shows that all techniques achieve satisfactory accuracy. It also shows that Finger-Count requires more mental demand than other techniques.

Design and evaluation of finger-count interaction: Combining multitouch gestures and menus

Selecting commands on multi-touch displays is still a challenging problem. While a number of gestural vocabularies have been proposed, these are generally restricted to one or two fingers or can be difficult to learn. We introduce Finger-Count gestures, a coherent set of multi-finger and two-handed gestures. Finger-Count gestures are simple, robust, expressive and fast to perform. In order to make these gestures self-revealing and easy to learn, we propose the Finger-Count menu, a menu technique and teaching method for implicitly learning Finger-Count gestures. We discuss the properties, advantages and limitations of Finger-Count interaction from the gesture and menu technique perspectives as well as its integration into three applications. We present alternative designs to increase the number of commands and to enable multi-user scenarios. Following a study which shows that Finger-Count is as easy to learn as radial menus, we report the results of an evaluation investigating which gestures are easier to learn and which finger chords people prefer. Finally, we present Finger-Count for in-the-air gestures. Thereby, the same gesture set can be used from a distance as well as when touching the surface.

PalmSpace: continuous around-device gestures vs. multitouch for 3D rotation tasks on mobile devices

Rotating 3D objects is a difficult task on mobile devices, because the task requires 3 degrees of freedom and (multi-)touch input only allows for an indirect mapping. We propose a novel style of mobile interaction based on mid-air gestures in proximity of the device to increase the number of DOFs and alleviate the limitations of touch interaction with mobile devices. While one hand holds the device, the other hand performs mid-air gestures in proximity of the device to control 3D objects on the mobile devices screen. A flat hand pose defines a virtual surface which we refer to as the PalmSpace for precise and intuitive 3D rotations. We constructed several hardware prototypes to test our interface and to simulate possible future mobile devices equipped with depth cameras. We conducted a user study to compare 3D rotation tasks using the most promising two designs for the hand location during interaction -- behind and beside the device -- with the virtual trackball, which is the current state-of-art technique for orientation manipulation on touch-screens. Our results show that both variants of PalmSpace have significantly lower task completion times in comparison to the virtual trackball.

Leaf Menus: Linear Menus with Stroke Shortcuts for Small Handheld Devices

This paper presents Leaf menu, a new type of contextual linear menu that supports curved gesture shortcuts. By providing an alternative to keyboard shortcuts, the Leaf menus can be used for the selection of commands on tabletops, but its key benefit is its adequacy to small handheld touchscreen devices (PDA, Smartphone). Indeed Leaf menus define a compact and known layout inherited from linear menus, they support precise finger interaction, they manage occlusion and they can be used in close proximity to the screen borders. Moreover, by providing stroke shortcuts, they favour the selection of frequent commands in expert mode and make eye-free selection possible.

Wave menus: improving the novice mode of hierarchical marking menus

We present Wave menus, a variant of multi-stroke marking menus designed for improving the novice mode of marking while preserving their efficiency in the expert mode of marking. Focusing on the novice mode, a criteria-based analysis of existing marking menus motivates the design of Wave menus. Moreover a user experiment is presented that compares four hierarchical marking menus in novice mode. Results show that Wave and compound-stroke menus are significantly faster and more accurate than multi-stroke menus in novice mode, while it has been shown that in expert mode the multistroke menus and therefore the Wave menus outperform the compound-stroke menus. Wave menus also require significantly less screen space than compound-stroke menus. As a conclusion, Wave menus offer the best performance for both novice and expert modes in comparison with existing multi-level marking menus, while requiring less screen space than compound-stroke menus.