Johan Kildal

Opportunities and Challenges for Data Physicalization

Physical representations of data have existed for thousands of years. Yet it is now that advances in digital fabrication, actuated tangible interfaces, and shape-changing displays are spurring an emerging area of research that we call Data Physicalization. It aims to help people explore, understand, and communicate data using computer-supported physical data representations. We call these representations physicalizations, analogously to visualizations -- their purely visual counterpart. In this article, we go beyond the focused research questions addressed so far by delineating the research area, synthesizing its open challenges and laying out a research agenda.

Kinetic device: designing interactions with a deformable mobile interface

We introduce the user-centered research that we are conducting using functional deformable research prototypes. This work has recently crystallized in the demonstration of the Nokia Kinetic Device (figure 1). In the large design space that opens before us around deformable user interfaces (DUIs), we have chosen to focus on mobile personal interfaces. We aim to investigate how human factors should influence the transition from rigid to deformable hardware. In this paper, we propose the topics that a research agenda should cover, and we discuss our research methodology. We also describe the functional deformable research prototype (called Kinetic DUI-RP) that we are using to conduct our research. Finally, we present an initial set of design guidelines that future research will develop further.

3D-press: haptic illusion of compliance when pressing on a rigid surface

This paper reports a new intramodal haptic illusion. This illusion involves a person pressing on a rigid surface and perceiving that the surface is compliant, i.e. perceiving that the contact point displaces into the surface. The design process, method and conditions used to create this illusion are described in detail. A user study is also reported in which all participants using variants of the basic method experienced the illusion, demonstrating the effectiveness of the method. This study also offers an initial indication of the mechanical dimensions of illusory compliance that could be manipulated by varying the stimuli presented to the users. This method could be used to augment touch interaction with mobile devices, transcending the rigid two-dimensional tangible surface (touch display) currently found on them.

Feeling it: the roles of stiffness, deformation range and feedback in the control of deformable ui

There has been little discussion on how the materials used to create deformable devices, and the subsequent interactions, might influence user performance and preference. In this paper we evaluated how the stiffness and required deformation extent (bending up and down bimanually) of mobile phone-shaped deformable devices influenced how precisely participants were able to move to and maintain target extents of deformation (bend). Given the inherent haptic feedback available from deforming devices (over rigid devices), we also compared performance with, and without, external visual feedback. User perception and preference regarding the different devices were also elicited. Results show that, while device stiffness did not significantly affect task performance, user comfort and preferences were strongly in favour of softer materials (0.45 N·m/rad) and moderate amounts of deformation. Removing external visual feedback led to less precise user input, but inaccuracy remained low enough to suggest non-visual interaction with deformable devices is feasible.

Twisting touch: combining deformation and touch as input within the same interaction cycle on handheld devices

We present a study that investigates the potential of combining, within the same interaction cycle, deformation and touch input in a handheld device. Using a flexible, input-only device connected to an external display, we compared a multitouch input technique and two hybrid deformation-plus-touch input techniques (bending and twisting the device, plus either front- or back-touch), in an image-docking task. We compared and analyzed the performance (completion time) and user experience (UX) obtained in each case, using multiple assessment metrics. We found that combining device deformation with front-touch produced the best UX. All the interaction techniques showed the same efficiency in task completion. This was a surprising finding, since multitouch (an integral input technique) was expected to be the most efficient technique in an image docking task (an interaction in an integral perceptual space). We discuss these findings in relation to self-reported qualitative data and observed interaction-procedure metrics. We found that the interaction procedures with the hybrid techniques were more sequential but also more paced. These findings suggest that the benefits of deformation input can still be observed when deformation and touch are combined in an input device.

Squeezy bracelet: designing a wearable communication device for tactile interaction

While smartphones are increasing in size and complex features, new form factors for simple communication devices are emerging. In this paper, we present the design process for a wrist worn communication device, which enables the user to send text messages over a paired mobile phone. The process includes concept design, user evaluation, design iteration, prototype implementation, and evaluation of alternative interaction techniques. Our particular focus is towards the use of naturally tactile interfaces in a wearable wristband form factor. We present how users perceive deformable communication device concepts and two alternative squeeze based interaction techniques.

Kooboh: variable tangible properties in a handheld haptic-illusion box

Kooboh is a handheld, tangible user interface (TUI) that can display various mechanical properties to the hand that grasps it. Made of rigid material, its cuboid shape and size never change. However, when manually squeezed or pressed, the user perceives in the hand that the object is compliant and deformable. This is achieved through a non-visual haptic illusion that can be controlled by software. With this technique, a TUI can display different mechanical feelings throughout the interaction cycle. This can greatly enhance the experience of handling and manipulating TUIs that, otherwise, only present one state: the feeling of the real physical object.

Interacting with deformable user interfaces: effect of material stiffness and type of deformation gesture

Deformable User Interfaces (DUIs) are increasingly being proposed for new tangible and organic interaction metaphors and techniques. To design DUIs, it is necessary to understand how deforming different materials manually using different gestures affects performance and user experience. In the study reported in this paper, three DUIs made of deformable materials with different levels of stiffness were used in navigation tasks that required bending and twisting the interfaces. Discrete and continuous deformation gestures were used in each case. Results showed that the stiffness of the material and the type of gesture affected performance and user experience in complex ways, but with a pervading pattern: using discrete gestures in very short navigation distances and continuous gestures otherwise, plus using lower-stiffness materials in every case, was beneficial in terms of performance and user experience.

Perceived physicality in audio-enhanced force input

This paper investigates how the perceived physicality of the action of applying force with a finger on a rigid surface (such as on a force-sensing touch screen) can be enhanced using real-time synthesized audio feedback. A selection of rich and evocative audio designs was used. Additionally, audio-tactile cross-modal integration was encouraged, by observing that the main rules of multisensory integration were supported. The study conducted showed that richness of perceived physicality increased considerably, mostly in its auditory expression (what pressing sounded like). In addition, in many instances it was observed that the haptic expression of physicality also increased (what pressing felt like), including some perception of compliance. This last result was particularly interesting as it showed that audio-tactile cross-modal integration might be present.

Orientation inquiry: a new haptic interaction technique for non-visual pedestrian navigation

Current mobile navigation systems often require visual attention. This may lead to both inconvenient and unsafe use while walking. In this paper, we are introducing orientation inquiry, a new haptic interaction technique for non-visual pedestrian navigation. In a pilot experiment, the orientation inquiry technique was compared to tactile icons used as vibration patterns indicating the direction of travel. The results suggest that both techniques are suitable for navigation, but the participants preferred orientation inquiry to tactile icons.