Krysta Chauncey

Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines

Because functional near-infrared spectroscopy (fNIRS) eases many of the restrictions of other brain sensors, it has potential to open up new possibilities for HCI research. From our experience using fNIRS technology for HCI, we identify several considerations and provide guidelines for using fNIRS in realistic HCI laboratory settings. We empirically examine whether typical human behavior (e.g. head and facial movement) or computer interaction (e.g. keyboard and mouse usage) interfere with brain measurement using fNIRS. Based on the results of our study, we establish which physical behaviors inherent in computer usage interfere with accurate fNIRS sensing of cognitive state information, which can be corrected in data analysis, and which are acceptable. With these findings, we hope to facilitate further adoption of fNIRS brain sensing technology in HCI research.

Effects of stimulus font and size on masked repetition priming: An event-related potentials (ERP) investigation

The size and font of target words were manipulated in a masked repetition priming paradigm with ERP recordings. Repetition priming effects were found in four ERP components: the N/P150, N250, P325, and N400. Neither a change in font nor a change in size across prime and target were found to affect repetition priming in the N250, P325, and N400 components. Changing font was, however, found to affect repetition priming in the N/P150 component, while the interaction between repetition priming and size was not significant in this component. These results confirm our interpretation of the N/P150 as a component sensitive to feature-level processing, and suggest that the type of prelexical and lexical processing reflected in the N250, P325, and N400 components is performed on representations that are invariant to changes in both font and size.

Code-switching effects in bilingual word recognition: A masked priming study with event-related potentials ☆

Two experiments tested language switching effects with bilingual participants in a priming paradigm with masked primes (duration of 50ms in Experiment 1 and 100ms in Experiment 2). Participants had to monitor target words for animal names, and ERPs were recorded to critical (non-animal) words in L1 and L2 primed by unrelated words from the same or the other language. Both experiments revealed language priming (switching) effects that depended on target language. For target words in L1, most of the language switch effect appeared in the N400 ERP component, with L2 primes generating a more negative going wave than L1 primes. For L2 target words, on the other hand, the effects of a language switch appeared mainly in an earlier ERP component (N250) peaking at approximately 250ms post-target onset, and showing greater negativity following an L1 prime than an L2 prime. This is the first evidence for fast-acting language-switching effects occurring in the absence of overt task switching.

Distinguishing Difficulty Levels with Non-invasive Brain Activity Measurements

Passive brain-computer interfaces are designed to use brain activity as an additional input, allowing the adaptation of the interface in real time according to the users mental state. The goal of the present study is to distinguish between different levels of game difficulty using non-invasive brain activity measurement with functional near-infrared spectroscopy (fNIRS). The study is designed to lead to adaptive interfaces that respond to the users brain activity in real time. Nine subjects played two levels of the game Pacman while their brain activity was measured using fNIRS. Statistical analysis and machine learning classification results show that we can discriminate well between subjects playing or resting, and distinguish between the two levels of difficulty with some success. In contrast to most previous fNIRS studies which only distinguish brain activity from rest, we attempt to tell apart two levels of brain activity, and our results show potential for using fNIRS in an adaptive game or user interface.

Combining Electroencephalograph and Functional Near Infrared Spectroscopy to Explore Users' Mental Workload

We discuss the physiological metrics that can be measured with electroencephalography (EEG) and functional near infrared spectroscopy (fNIRs). We address the functional and practical limitations of each device, and technical issues to be mindful of when combining the devices. We also present machine learning methods that can be used on concurrent recordings of EEG and fNIRs data. We discuss an experiment that combines fNIRs and EEG to measure a range of user states that are of interest in HCI. While our fNIRS machine learning results showed promise for the measurement of workload states in HCI, our EEG results indicate that more research must be done in order to combine these two devices in practice.

Sensing cognitive multitasking for a brain-based adaptive user interface

Multitasking has become an integral part of work environments, even though people are not well-equipped cognitively to handle numerous concurrent tasks effectively. Systems that support such multitasking may produce better performance and less frustration. However, without understanding the users internal processes, it is difficult to determine optimal strategies for adapting interfaces, since all multitasking activity is not identical. We describe two experiments leading toward a system that detects cognitive multitasking processes and uses this information as input to an adaptive interface. Using functional near-infrared spectroscopy sensors, we differentiate four cognitive multitasking processes. These states cannot readily be distinguished using behavioral measures such as response time, accuracy, keystrokes or screen contents. We then present our human-robot system as a proof-of-concept that uses real-time cognitive state information as input and adapts in response. This prototype system serves as a platform to study interfaces that enable better task switching, interruption management, and multitasking.

Primed picture naming within and across languages: an ERP investigation.

In two experiments, while event-related potentials (ERPs) were recorded, participants named picture targets that were preceded by masked word primes that corresponded either to the name of the picture target or to an unrelated picture name. Experiment 1 showed significant priming effects in the ERP waveforms, free from articulator artifact, starting as early as 200 msec post target onset. Possible loci of these priming effects were proposed within the framework of generic interactive activation models of word recognition and picture naming. These were grouped into three main components: object-specific structural representations, amodal semantic representations, and word-specific phonological and articulatory representations. Experiment 2 provided an initial test of the possible role of each of these components by comparing within-language repetition priming with priming from translation equivalents in bilingual participants. The early and widespread effects of noncognate translation primes in L1 on picture naming in L2 point to object-specific and amodal semantic representations as the principal loci of priming effects obtained with masked word primes and picture targets.

The role of subjective frequency in language switching: An ERP investigation using masked priming

Two experiments examined the nature of language-switching effects in a priming paradigm with event-related brain potential (ERP) recordings. primes and targets were always unrelated words but could be either from the same or different languages (Experiment 1) or from the same or a different frequency range (Experiment 2). Effects of switching language across prime and target differed as a function of the direction of the switch and prime duration in Experiment 1. Effects tended to be stronger with 100-ms prime durations than with 50-ms durations, and the expected pattern of greater negativity in the switch condition appeared earlier when primes were in L1 and targets in L2 than vice versa. Experiment 2 examined whether these language-switching effects could be due to differences in the subjective frequency of words in a bilingual’s two languages, by testing a frequency-switching manipulation within the L1. Effects of frequency switching were evident in the ERP waveforms, but the pattern did not resemble the language-switching effects, therefore suggesting that different mechanisms are at play.

From Brain Signals to Adaptive Interfaces: Using fNIRS in HCI

Functional near-infrared spectroscopy (fNIRS) is an emerging non-invasive, lightweight imaging tool which can measure blood oxygenation levels in the brain. In this chapter, we describe the fNIRS device and its potential within the realm of human-computer interaction (HCI). We discuss research that explores the kinds of states that can be measured with fNIRS, and we describe initial research and prototypes that can use this objective, real time information about users’ states as input to adaptive user interfaces.

Your Brain, Your Computer, and You

Passive brain-computer interfaces increase the bandwidth from user to computer in new and uniquely powerful ways.