Audrey Girouard

Reality-based interaction: a framework for post-WIMP interfaces

We are in the midst of an explosion of emerging human-computer interaction techniques that redefine our understanding of both computers and interaction. We propose the notion of Reality-Based Interaction (RBI) as a unifying concept that ties together a large subset of these emerging interaction styles. Based on this concept of RBI, we provide a framework that can be used to understand, compare, and relate current paths of recent HCI research as well as to analyze specific interaction designs. We believe that viewing interaction through the lens of RBI provides insights for design and uncovers gaps or opportunities for future research.

PaperPhone: understanding the use of bend gestures in mobile devices with flexible electronic paper displays

Flexible displays potentially allow for interaction styles that resemble those used in paper documents. Bending the display, e.g., to page forward, shows particular promise as an interaction technique. In this paper, we present an evaluation of the effectiveness of various bend gestures in executing a set of tasks with a flexible display. We discuss a study in which users designed bend gestures for common computing actions deployed on a smartphone-inspired flexible E Ink prototype called PaperPhone. We collected a total of 87 bend gesture pairs from ten participants and their appropriateness over twenty actions in five applications. We identified six most frequently used bend gesture pairs out of 24 unique pairs. Results show users preferred bend gestures and bend gesture pairs that were conceptually simpler, e.g., along one axis, and less physically demanding. There was a strong agreement among participants to use the same three pairs in applications: (1) side of display, up/down (2) top corner, up/down (3) bottom corner, up/down. For actions with a strong directional cue, we found strong consensus on the polarity of the bend gestures (e.g., navigating left is performed with an upwards bend gesture, navigating right, downwards). This implies that bend gestures that take directional cues into account are likely more natural to users.

Brain measurement for usability testing and adaptive interfaces: an example of uncovering syntactic workload with functional near infrared spectroscopy

A well designed user interface (UI) should be transparent, allowing users to focus their mental workload on the task at hand. We hypothesize that the overall mental workload required to perform a task using a computer system is composed of a portion attributable to the difficulty of the underlying task plus a portion attributable to the complexity of operating the user interface. In this regard, we follow Shneidermans theory of syntactic and semantic components of a UI. We present an experiment protocol that can be used to measure the workload experienced by users in their various cognitive resources while working with a computer. We then describe an experiment where we used the protocol to quantify the syntactic workload of two user interfaces. We use functional near infrared spectroscopy, a new brain imaging technology that is beginning to be used in HCI. We also discuss extensions of our techniques to adaptive interfaces.

Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines

Because functional near-infrared spectroscopy (fNIRS) eases many of the restrictions of other brain sensors, it has potential to open up new possibilities for HCI research. From our experience using fNIRS technology for HCI, we identify several considerations and provide guidelines for using fNIRS in realistic HCI laboratory settings. We empirically examine whether typical human behavior (e.g. head and facial movement) or computer interaction (e.g. keyboard and mouse usage) interfere with brain measurement using fNIRS. Based on the results of our study, we establish which physical behaviors inherent in computer usage interfere with accurate fNIRS sensing of cognitive state information, which can be corrected in data analysis, and which are acceptable. With these findings, we hope to facilitate further adoption of fNIRS brain sensing technology in HCI research.

TeleHuman: effects of 3d perspective on gaze and pose estimation with a life-size cylindrical telepresence pod

In this paper, we present TeleHuman, a cylindrical 3D display portal for life-size human telepresence. The TeleHuman 3D videoconferencing system supports 360 degree motion parallax as the viewer moves around the cylinder and optionally, stereoscopic 3D display of the remote person. We evaluated the effect of perspective cues on the conveyance of nonverbal cues in two experiments using a one-way telecommunication version of the system. The first experiment focused on how well the system preserves gaze and hand pointing cues. The second experiment evaluated how well the system conveys 3D body postural information. We compared 3 perspective conditions: a conventional 2D view, a 2D view with 360 degree motion parallax, and a stereoscopic view with 360 degree motion parallax. Results suggest the combined presence of motion parallax and stereoscopic cues significantly improved the accuracy with which participants were able to assess gaze and hand pointing cues, and to instruct others on 3D body poses. The inclusion of motion parallax and stereoscopic cues also led to significant increases in the sense of social presence and telepresence reported by participants.

DISCRIMINATION OF MENTAL WORKLOAD LEVELS IN HUMAN SUBJECTS WITH FUNCTIONAL NEAR-INFRARED SPECTROSCOPY

We have applied functional near-infrared spectroscopy (fNIRS) to the human forehead to distinguish different levels of mental workload on the basis of hemodynamic changes occurring in the prefrontal cortex. We report data on 3 subjects from a protocol involving 3 mental workload levels based on to working memory tasks. To quantify the potential of fNIRS for mental workload discrimination, we have applied a 3-nearest neighbor classification algorithm based on the amplitude of oxyhemoglobin (HbO2) and deoxyhemoglobin (HbR) concentration changes associated with the working memory tasks. We have found classification success rates in the range of 44%–72%, which are significantly higher than the corresponding chance level (for random data) of 19.1%. This work shows the potential of fNIRS for mental workload classification, especially when more parameters (rather than just the amplitude of concentration changes used here) and more sophisticated classification algorithms (rather than the simple 3-nearest neighbor algorithm used here) are considered and optimized for this application.

Reality-based interaction: unifying the new generation of interaction styles

We are in the midst of an explosion of emerging human-computer interaction techniques that have redefined our understanding of both computers and interaction. We propose the notion of Reality-Based Interaction (RBI) as a unifying concept that ties together a large subset of these emerging interaction styles. Through RBI we are attempting to provide a framework that can be used to understand, compare, and relate current paths of HCI research. Viewing interaction through the lens of RBI can provide insights for designers and allows us to find gaps or opportunities for future development. Furthermore, we are using RBI to develop new evaluation techniques for features of emerging interfaces that are currently unquantifiable.

Pointable: an in-air pointing technique to manipulate out-of-reach targets on tabletops

Selecting and moving digital content on interactive tabletops often involves accessing the workspace beyond arms reach. We present Pointable, an in-air, bimanual perspective-based interaction technique that augments touch input on a tabletop for distant content. With Pointable, the dominant hand selects remote targets, while the non-dominant hand can scale and rotate targets with a dynamic C/D gain. We conducted 3 experiments; the first showed that pointing at a distance using Pointable has a Fitts law throughput comparable to that of a mouse. In the second experiment, we found that Pointable had the same performance as multi-touch input in a resize, rotate and drag task. In a third study, we observed that when given the choice, over 75% of participants preferred to use Pointable over multi-touch for target manipulation. In general, Pointable allows users to manipulate out-of-reach targets, without loss of performance, while minimizing the need to lean, stand up, or involve collocated collaborators.

Distinguishing Difficulty Levels with Non-invasive Brain Activity Measurements

Passive brain-computer interfaces are designed to use brain activity as an additional input, allowing the adaptation of the interface in real time according to the users mental state. The goal of the present study is to distinguish between different levels of game difficulty using non-invasive brain activity measurement with functional near-infrared spectroscopy (fNIRS). The study is designed to lead to adaptive interfaces that respond to the users brain activity in real time. Nine subjects played two levels of the game Pacman while their brain activity was measured using fNIRS. Statistical analysis and machine learning classification results show that we can discriminate well between subjects playing or resting, and distinguish between the two levels of difficulty with some success. In contrast to most previous fNIRS studies which only distinguish brain activity from rest, we attempt to tell apart two levels of brain activity, and our results show potential for using fNIRS in an adaptive game or user interface.

Combining Electroencephalograph and Functional Near Infrared Spectroscopy to Explore Users' Mental Workload

We discuss the physiological metrics that can be measured with electroencephalography (EEG) and functional near infrared spectroscopy (fNIRs). We address the functional and practical limitations of each device, and technical issues to be mindful of when combining the devices. We also present machine learning methods that can be used on concurrent recordings of EEG and fNIRs data. We discuss an experiment that combines fNIRs and EEG to measure a range of user states that are of interest in HCI. While our fNIRS machine learning results showed promise for the measurement of workload states in HCI, our EEG results indicate that more research must be done in order to combine these two devices in practice.