Helena M. Mentis

Instructing people for training gestural interactive systems

Entertainment and gaming systems such as the Wii and XBox Kinect have brought touchless, body-movement based interfaces to the masses. Systems like these enable the estimation of movements of various body parts from raw inertial motion or depth sensor data. However, the interface developer is still left with the challenging task of creating a system that recognizes these movements as embodying meaning. The machine learning approach for tackling this problem requires the collection of data sets that contain the relevant body movements and their associated semantic labels. These data sets directly impact the accuracy and performance of the gesture recognition system and should ideally contain all natural variations of the movements associated with a gesture. This paper addresses the problem of collecting such gesture datasets. In particular, we investigate the question of what is the most appropriate semiotic modality of instructions for conveying to human subjects the movements the system developer needs them to perform. The results of our qualitative and quantitative analysis indicate that the choice of modality has a significant impact on the performance of the learnt gesture recognition system; particularly in terms of correctness and coverage.

Touchless interaction in surgery

Touchless interaction with medical images lets surgeons maintain sterility during surgical procedures.

On the naturalness of touchless: Putting the “interaction” back into NUI

Normans critique is indicative of the issue that while using the word natural might have become natural, it is coming at a cost. In other words, precisely because the notion of naturalness has become so commonplace in the scientific lexicon of HCI, so it is becoming increasingly important, it seems that there is a critical examination of the conceptual work being performed when it is used. There is a need to understand the key assumptions implicit within it and how these frame approaches to design and engineering in particular ways. A second significant element of this perspective comes from Wittgenstein, and his claim that, through action, people create shared meanings with others, and these shared meanings are the essential common ground that enable individual perception to be cohered into socially organized, understood, and coordinated experiences.

Articulating common ground in cooperative work: content and process

We study the development of common ground in an emergency management planning task. Twelve three-person multi-role teams performed the task with a paper prototype in a controlled setting; each team completed three versions of the task. We use converging measures to document the development of common ground in the teams and present an in-depth analysis of the characteristics of the common ground development process. Our findings indicate that in complex collaborative work, process common ground increases, thus diminishing the need for acts like information querying or strategy discussions about how to organize the collaborative activities. However, content common ground is created and tested throughout the three runs; in fact dialogue acts used to clarify this content increase over time. Discussion of the implications of these findings for the theory of common ground and the design of collaborative systems follows.

Interaction proxemics and image use in neurosurgery

Within medical settings there is a growing interest in exploring touchless interaction technologies. The primary motivation here is to avoid contact during interaction with data so as to maintain asepsis. However, there is another important property of touchless interaction that has significant implications for their use within such settings -- namely that interaction behaviour is spatially distal from the device being interacted with. To further understand these implications we present fieldwork observations of work practice in neurosurgery theatres. Drawing on the notion of interaction proxemics and the theory of F-formations, our analysis articulates the spatial organization of collaborative work practices and interaction in these settings. From this understanding of spatial practices, we discuss opportunities and difficulties relating to the design of touchless interaction technologies for in surgical settings.

Supporting content and process common ground in computer-supported teamwork

We build on our prior work with computer-supported teams performing a complex decision-making task on maps, where the distinction between content and process common ground is proposed. In this paper we describe a distributed geo-collaboration software prototype. The system design rationale was gleaned from fieldwork, literature on team cognition, and an earlier lab study introducing a reference task with face-to-face teams. We report on a controlled experiment that evaluates this design rationale. Distinct sets of measures show that that the prototype supported both content and process common ground, offsetting the costs imposed by the distributed setting. We interpret the results in relation to prior work on common ground and draw implications for moving beyond current models of sharing and coordination.

Supporting common ground and awareness in emergency management planning: A design research project

We present a design research project on knowledge sharing and activity awareness in distributed emergency management planning. In three experiments we studied groups using three different prototypes, respectively: a paper-prototype in a collocated work setting, a first software prototype in a distributed setting, and a second, enhanced software prototype in a distributed setting. In this series of studies we tried to better understand the processes of knowledge sharing and activity awareness in complex cooperative work by developing and investigating new tools that can support these processes. We explicate the design rationale behind each prototype and report the results of each experiment investigating it. We discuss how the results from each prototyping phase brought us closer to defining properties of a system that facilitate the sharing and awareness of both content and process knowledge. Our designs enhanced aspects of distributed group performance, in some respects beyond that of comparable face-to-face groups.

Interactional Order and Constructed Ways of Seeing with Touchless Imaging Systems in Surgery

While surgical practices are increasingly reliant on a range of digital imaging technologies, the ability for clinicians to interact and manipulate these digital representations in the operating theatre using traditional touch based interaction devices is constrained by the need to maintain sterility. To overcome these concerns with sterility, a number of researchers are have been developing ways of enabling interaction in the operating theatre using touchless interaction techniques such as gesture and voice to allow clinicians control of the systems. While there have been important technical strides in the area, there has been little in the way of understanding the use of these touchless systems in practice. With this in mind we present a touchless system developed for use during vascular surgery. We deployed the system in the endovascular suite of a large hospital for use in the context of real procedures. We present findings from a study of the system in use focusing on how, with touchless interaction, the visual resources were embedded and made meaningful in the collaborative practices of surgery. In particular we discuss the importance of direct and dynamic control of the images by the clinicians in the context of talk and in the context of other artefact use as well as the work performed by members of the clinical team to make themselves sensable by the system. We discuss the broader implications of these findings for how we think about the design, evaluation and use of these systems.

Using TouchPad pressure to detect negative affect

Humans naturally use behavioral cues in their interactions with other humans. The Media Equation proposes that these same cues are directed towards media, including computers. It is probable that detection of these cues by a computer during run-time could improve usability design and analysis. A preliminary experiment testing one of these cues, Synaptics TouchPad pressure, shows that behavioral cues can be used as a critical incident indicator by detecting negative affect.

Imaging the body: embodied vision in minimally invasive surgery

Recent years have seen the possibilities of new imaging and interaction technologies for minimally invasive surgery such as touchless interaction and high definition renderings of three-dimensional anatomy. With this paper we take a step back to review the historical introduction and assimilation of imaging technologies in the surgical theatre in parallel with the productive and cross-referential nature of surgical practice and image use. We present findings from a field study of image use during neurosurgery where we see that the work to see medical images is highly constructed and embodied with the action of manipulating the body. This perspective lends itself to a discussion of the directions for new imaging interaction technologies.