Leanne M. Hirshfield

Reality-based interaction: a framework for post-WIMP interfaces

We are in the midst of an explosion of emerging human-computer interaction techniques that redefine our understanding of both computers and interaction. We propose the notion of Reality-Based Interaction (RBI) as a unifying concept that ties together a large subset of these emerging interaction styles. Based on this concept of RBI, we provide a framework that can be used to understand, compare, and relate current paths of recent HCI research as well as to analyze specific interaction designs. We believe that viewing interaction through the lens of RBI provides insights for design and uncovers gaps or opportunities for future research.

Brain measurement for usability testing and adaptive interfaces: an example of uncovering syntactic workload with functional near infrared spectroscopy

A well designed user interface (UI) should be transparent, allowing users to focus their mental workload on the task at hand. We hypothesize that the overall mental workload required to perform a task using a computer system is composed of a portion attributable to the difficulty of the underlying task plus a portion attributable to the complexity of operating the user interface. In this regard, we follow Shneidermans theory of syntactic and semantic components of a UI. We present an experiment protocol that can be used to measure the workload experienced by users in their various cognitive resources while working with a computer. We then describe an experiment where we used the protocol to quantify the syntactic workload of two user interfaces. We use functional near infrared spectroscopy, a new brain imaging technology that is beginning to be used in HCI. We also discuss extensions of our techniques to adaptive interfaces.

Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines

Because functional near-infrared spectroscopy (fNIRS) eases many of the restrictions of other brain sensors, it has potential to open up new possibilities for HCI research. From our experience using fNIRS technology for HCI, we identify several considerations and provide guidelines for using fNIRS in realistic HCI laboratory settings. We empirically examine whether typical human behavior (e.g. head and facial movement) or computer interaction (e.g. keyboard and mouse usage) interfere with brain measurement using fNIRS. Based on the results of our study, we establish which physical behaviors inherent in computer usage interfere with accurate fNIRS sensing of cognitive state information, which can be corrected in data analysis, and which are acceptable. With these findings, we hope to facilitate further adoption of fNIRS brain sensing technology in HCI research.

DISCRIMINATION OF MENTAL WORKLOAD LEVELS IN HUMAN SUBJECTS WITH FUNCTIONAL NEAR-INFRARED SPECTROSCOPY

We have applied functional near-infrared spectroscopy (fNIRS) to the human forehead to distinguish different levels of mental workload on the basis of hemodynamic changes occurring in the prefrontal cortex. We report data on 3 subjects from a protocol involving 3 mental workload levels based on to working memory tasks. To quantify the potential of fNIRS for mental workload discrimination, we have applied a 3-nearest neighbor classification algorithm based on the amplitude of oxyhemoglobin (HbO2) and deoxyhemoglobin (HbR) concentration changes associated with the working memory tasks. We have found classification success rates in the range of 44%–72%, which are significantly higher than the corresponding chance level (for random data) of 19.1%. This work shows the potential of fNIRS for mental workload classification, especially when more parameters (rather than just the amplitude of concentration changes used here) and more sophisticated classification algorithms (rather than the simple 3-nearest neighbor algorithm used here) are considered and optimized for this application.

Reality-based interaction: unifying the new generation of interaction styles

We are in the midst of an explosion of emerging human-computer interaction techniques that have redefined our understanding of both computers and interaction. We propose the notion of Reality-Based Interaction (RBI) as a unifying concept that ties together a large subset of these emerging interaction styles. Through RBI we are attempting to provide a framework that can be used to understand, compare, and relate current paths of HCI research. Viewing interaction through the lens of RBI can provide insights for designers and allows us to find gaps or opportunities for future development. Furthermore, we are using RBI to develop new evaluation techniques for features of emerging interfaces that are currently unquantifiable.

Distinguishing Difficulty Levels with Non-invasive Brain Activity Measurements

Passive brain-computer interfaces are designed to use brain activity as an additional input, allowing the adaptation of the interface in real time according to the users mental state. The goal of the present study is to distinguish between different levels of game difficulty using non-invasive brain activity measurement with functional near-infrared spectroscopy (fNIRS). The study is designed to lead to adaptive interfaces that respond to the users brain activity in real time. Nine subjects played two levels of the game Pacman while their brain activity was measured using fNIRS. Statistical analysis and machine learning classification results show that we can discriminate well between subjects playing or resting, and distinguish between the two levels of difficulty with some success. In contrast to most previous fNIRS studies which only distinguish brain activity from rest, we attempt to tell apart two levels of brain activity, and our results show potential for using fNIRS in an adaptive game or user interface.

Combining Electroencephalograph and Functional Near Infrared Spectroscopy to Explore Users' Mental Workload

We discuss the physiological metrics that can be measured with electroencephalography (EEG) and functional near infrared spectroscopy (fNIRs). We address the functional and practical limitations of each device, and technical issues to be mindful of when combining the devices. We also present machine learning methods that can be used on concurrent recordings of EEG and fNIRs data. We discuss an experiment that combines fNIRs and EEG to measure a range of user states that are of interest in HCI. While our fNIRS machine learning results showed promise for the measurement of workload states in HCI, our EEG results indicate that more research must be done in order to combine these two devices in practice.

Smart Blocks: a tangible mathematical manipulative

We created Smart Blocks, an augmented mathematical manipulative that allows users to explore the concepts of volume and surface area of 3-dimensional (3D) objects. This interface supports physical manipulation for exploring spatial relationships and it provides continuous feedback for reinforcing learning. By leveraging the benefits of physicality with the advantages of digital information, this tangible interface provides an engaging environment for learning about surface area and volume of 3D objects.

This is your brain on interfaces: enhancing usability testing with functional near-infrared spectroscopy

This project represents a first step towards bridging the gap between HCI and cognition research. Using functional near-infrared spectroscopy (fNIRS), we introduce tech-niques to non-invasively measure a range of cognitive workload states that have implications to HCI research, most directly usability testing. We present a set of usability experiments that illustrates how fNIRS brain measurement provides information about the cognitive demands placed on computer users by different interface designs.

From Brain Signals to Adaptive Interfaces: Using fNIRS in HCI

Functional near-infrared spectroscopy (fNIRS) is an emerging non-invasive, lightweight imaging tool which can measure blood oxygenation levels in the brain. In this chapter, we describe the fNIRS device and its potential within the realm of human-computer interaction (HCI). We discuss research that explores the kinds of states that can be measured with fNIRS, and we describe initial research and prototypes that can use this objective, real time information about users’ states as input to adaptive user interfaces.