Dustin Freeman

KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera

KinectFusion enables a user holding and moving a standard Kinect camera to rapidly create detailed 3D reconstructions of an indoor scene. Only the depth data from Kinect is used to track the 3D pose of the sensor and reconstruct, geometrically precise, 3D models of the physical scene in real-time. The capabilities of KinectFusion, as well as the novel GPU-based pipeline are described in full. Uses of the core system for low-cost handheld scanning, and geometry-aware augmented reality and physics-based interactions are shown. Novel extensions to the core GPU pipeline demonstrate object segmentation and user interaction directly in front of the sensor, without degrading camera tracking or reconstruction. These extensions are used to enable real-time multi-touch interactions anywhere, allowing any planar or non-planar reconstructed physical surface to be appropriated for touch.

ShadowGuides: visualizations for in-situ learning of multi-touch and whole-hand gestures

We present ShadowGuides, a system for in-situ learning of multi-touch and whole-hand gestures on interactive surfaces. ShadowGuides provides on-demand assistance to the user by combining visualizations of the users current hand posture as interpreted by the system (feedback) and available postures and completion paths necessary to finish the gesture (feedforward). Our experiment compared participants learning gestures with ShadowGuides to those learning with video-based instruction. We found that participants learning with ShadowGuides remembered more gestures and expressed significantly higher preference for the help system.

Freehand pose-based Gestural Interaction: Studies and implications for interface design

Most work on Freehand Gestural Interaction has focused on high-energy expressive interfaces for expert user. In this work, we examine the use of hand poses in laid-back freehand gestural interactions for novice users and examine the factors that impact gesture performance. Through two Wizard-of-Oz studies, one leading to the other, we observe how novice users behave under relaxed conditions. The first study explores the ease of use of a pose-based hand gesture vocabulary in the context of a photo-browsing task, and examines some of the key factors that impact the performance of such pose based gestures. The second explores pose-based interaction techniques for widget manipulation tasks. These studies reveal that while hand poses have the potential to expand the vocabulary of gestures and are easy to recall and use, there are a number of issues that show up in actual performance related to inadvertent modifications in hand pose and hand trajectories. We summarize the implications of these findings for the design of pose-based freehand gestural interfaces, which we believe would be useful for both interaction designers and gesture recognition researchers.

Tweetris: a study of whole-body interaction during a public art event

We explore whole-body interaction with Tweetris, a game where two players competitively race to form Tetris shapes (tetrominos) with their body. We debuted Tweetris at an all-night, public art event, collecting 6000 winning body shapes made by more than 270 players. Tweetris employs a novel form of interaction cue we call a discretized silhouette: the mapping from physical continuous input is discretized to create a virtual body representation. Discretization creates an interesting set of properties: notably, players have a great deal of flexibility in how they create a given shape with their body. We classify and analyze successful player strategies as design input for whole body interaction, and present results showing how small differences in environment impacted player behaviour. We argue that our approach to eliciting and analyzing interaction in Tweetris has general utility to researchers and designers and we formalize it as the LoFi (Low-Fidelity) Elicitation Protocol.

The role of physical controllers in motion video gaming

Systems that detect the unaugmented human body allow players to interact without using a physical controller. But how is interaction altered by the absence of a physical input device? What is the impact on game performance, on a players expectation of their ability to control the game, and on their game experience? In this study, we investigate these issues in the context of a table tennis video game. The results show that the impact of holding a physical controller, or indeed of the fidelity of that controller, does not appear in simple measures of performance. Rather, the difference between controllers is a function of the responsiveness of the game being controlled, as well as other factors to do with expectations, real world game experience and social context.

Blending Art Events and HCI Research

We present experiences as artists and Human-Computer Interaction (HCI) researchers exhibiting an interactive artwork called Tweetris at a public event, and its simultaneous research evaluation. We describe the unique opportunities a public art event offered for achieving our research goals, then discuss three key challenges we encountered: tensions between creative and research goals before the event, ethical considerations during the event and in analysis, and obstacles complicating subsequent evaluation as the work has evolved. We offer observations throughout that are important to consider when conducting HCI research at public art events.

HCI: human-computer improvisation

This workshop explores the forms of improvisation that exist across various disciplines, how they can be observed empirically, how improvisation relates to technology and design, and how communities of improvisation exist and evolve. Through the use of these topics to stimulate discussion, along with group activities founded in theatre and music improvisation, we investigate how the study of improvisation can be used to inform contemporary HCI.

Tangible actions

We present Tangible Actions, an ad-hoc, just-in-time, visual programming by example language designed for large multitouch interfaces. With the design of Tangible Actions, we contribute a continually-created system of programming tokens that occupy the same space as the objects they act on. Tangible Actions are created by the gestural actions of the user, and they allow the user to reuse and modify their own gestures with a lower interaction cost than the original gesture. We implemented Tangible Actions in three different tabletop applications, and ran an informal evaluation. While we found that study participants generally liked and understood Tangible Actions, having the objects and the actions co-located can lead to visual and interaction clutter.

Supply Chain Requirements Engineering: A Simulated Reality Check

This paper presents a realistic understanding of the field of software requirement engineering of automotive spare parts dealers’ distribution and maintenance workshops services and spare parts supply chain management information systems. It attempts to elaborate elicitation techniques to get to actual requirements used by system analysts. These requirements establish needs of customers and users associated to the automotive parts, which can lead to desired software goal achievement of an organization. The magnitudes are also characterized by process interdependencies, interpersonal and inter-organizational conflicts and information uncertainties, and their interrelations. Problems are also described in this paper that occur in implementing a major organizational change initiative where various employee groups have different understandings of the rationale of the project and strategies intended to achieve its goals.

Improv Remix: Mixed-Reality Video Manipulation Using Whole-Body Interaction to Extend Improvised Theatre

Improv Remix is a mixed-reality system for live onstage video editing, using whole-body interaction, as an extension of improvised theatre. This work documents the process of analyzing an art form, then building technology that supports its extension. We tested the potential for video integration in improv in a exploratory workshop, then determined features and interaction techniques through iterative development with improvisors. We demonstrated the final iteration in a public showcase. Our contributions are (1) the documentation of the process, (2) a basic set of always-on interaction techniques tailored for performers standing adjacent to a large display, (3) methods to remix stage performance video, and, (4) a collection of creative use cases for the system, with an analysis of how the system extends improvisation.