Patricia A. Shewokis

Optical brain monitoring for operator training and mental workload assessment

An accurate measure of mental workload in human operators is a critical element of monitoring and adaptive aiding systems that are designed to improve the efficiency and safety of human-machine systems during critical tasks. Functional near infrared (fNIR) spectroscopy is a field-deployable non-invasive optical brain monitoring technology that provides a measure of cerebral hemodynamics within the prefrontal cortex in response to sensory, motor, or cognitive activation. In this paper, we provide evidence from two studies that fNIR can be used in ecologically valid environments to assess the: 1) mental workload of operators performing standardized (n-back) and complex cognitive tasks (air traffic control--ATC), and 2) development of expertise during practice of complex cognitive and visuomotor tasks (piloting unmanned air vehicles--UAV). Results indicate that fNIR measures are sensitive to mental task load and practice level, and provide evidence of the fNIR deployment in the field for its ability to monitor hemodynamic changes that are associated with relative cognitive workload changes of operators. The methods reported here provide guidance for the development of strategic requirements necessary for the design of complex human-machine interface systems and assist with assessments of human operator performance criteria.

Continuous monitoring of brain dynamics with functional near infrared spectroscopy as a tool for neuroergonomic research: empirical examples and a technological development

Functional near infrared spectroscopy (fNIRS) is a non-invasive, safe, and portable optical neuroimaging method that can be used to assess brain dynamics during skill acquisition and performance of complex work and everyday tasks. In this paper we describe neuroergonomic studies that illustrate the use of fNIRS in the examination of training-related brain dynamics and human performance assessment. We describe results of studies investigating cognitive workload in air traffic controllers, acquisition of dual verbal-spatial working memory skill, and development of expertise in piloting unmanned vehicles. These studies used conventional fNIRS devices in which the participants were tethered to the device while seated at a workstation. Consistent with the aims of mobile brain imaging (MoBI), we also describe a compact and battery-operated wireless fNIRS system that performs with similar accuracy as other established fNIRS devices. Our results indicate that both wired and wireless fNIRS systems allow for the examination of brain function in naturalistic settings, and thus are suitable for reliable human performance monitoring and training assessment.

Sliding-window motion artifact rejection for Functional Near-Infrared Spectroscopy

Functional Near-Infrared Spectroscopy (fNIR) is an optical brain monitoring technology that tracks changes in hemodynamic responses within the cortex. fNIR uses specific wavelengths of light, introduced at the scalp, to enable the noninvasive measurement of changes in the relative ratios of deoxygenated hemoglobin (deoxy-Hb) and oxygenated hemoglobin (oxy-Hb) during brain activity. This technology allows the design of portable, safe, affordable, noninvasive, and minimally intrusive monitoring systems that can be used to measure brain activity in natural environments, ambulatory and field conditions. However, for such applications fNIR signals can get prone to noise due to motion of the head. Improving signal quality and reducing noise, can be especially challenging for real time applications. Here, we study motion artifact related noise especially due to poor and changing sensor coupling. We have developed a simple and iterative method that can be used to automate the preprocessing of data to identify segments with such noise for exclusion and this method is also suitable for real time applications.

Assessment of Cognitive Neural Correlates for a Functional Near Infrared-Based Brain Computer Interface System

Functional Near Infrared Spectroscopy (fNIR) is a promising brain imaging technology that relies on optical techniques to detect changes of hemodynamic responses within the prefrontal cortex in response to sensory, motor, or cognitive activation. fNIR is safe, non-invasive, affordable, and highly portable. The objective of this study is to determine if biomarkers of neural activity generated by intentional cognitive activity, as measured by fNIR, can be used to communicate directly from the brain to a computer. A bar-size-control task based on a closed-loop system was designed and tested with 5 healthy subjects across two days. Comparisons of the average task and rest period oxygenation changes are significantly different (p<0.01). The average task completion time (reaching +90%) decreases with practice: day1 (mean 52.3 sec) and day2 (mean 39.1 sec). These preliminary results suggest that a closed-loop fNIR-based BCI can allow for a human-computer interaction with a mind switch task.

Monitoring expertise development during simulated UAV piloting tasks using optical brain imaging

An accurate assessment of mental workload and expertise level would help improve operational safety and efficacy of human computer interaction for aerospace applications. The current study utilized functional near-infrared spectroscopy (fNIR) to investigate the relationship of the hemodynamic response in the anterior prefrontal cortex to changes in mental workload, level of expertise, and task performance during learning of simulated unmanned aerial vehicle (UAV) piloting tasks. Results indicated that fNIR measures are correlated to task performance and subjective self-reported measures; and contained additional information that allowed categorizing learning phases. Level of expertise does appear to influence the hemodynamic response in the dorsolateral/ventrolateral prefrontal cortices. Since fNIR allows development of portable and wearable instruments, it has the potential to be deployed in future learning environments to personalize the training regimen and/or assess the effort of human operators in critical multitasking settings.

Functional near-infrared spectroscopy-based correlates of prefrontal cortical dynamics during a cognitive-motor executive adaptation task

This study investigated changes in brain hemodynamics, as measured by functional near infrared spectroscopy, during performance of a cognitive-motor adaptation task. The adaptation task involved the learning of a novel visuomotor transformation (a 60° counterclockwise screen-cursor rotation), which required inhibition of a prepotent visuomotor response. A control group experienced a familiar transformation and thus, did not face any executive challenge. Analysis of the experimental group hemodynamic responses revealed that the performance enhancement was associated with a monotonic reduction in the oxygenation level in the prefrontal cortex. This finding confirms and extends functional magnetic resonance imaging and electroencephalography studies of visuomotor adaptation and learning. The changes in prefrontal brain activation suggest an initial recruitment of frontal executive functioning to inhibit prepotent visuomotor mappings followed by a progressive de-recruitment of the same prefrontal regions. The prefrontal hemodynamic changes observed in the experimental group translated into enhanced motor performance revealed by a reduction in movement time, movement extent, root mean square error and the directional error. These kinematic adaptations are consistent with the acquisition of an internal model of the novel visuomotor transformation. No comparable change was observed in the control group for either the hemodynamics or for the kinematics. This study (1) extends our understanding of the frontal executive processes from the cognitive to the cognitive-motor domain and (2) suggests that optical brain imaging can be employed to provide hemodynamic based-biomarkers to assess and monitor the level of adaptive cognitive-motor performance.

THE EVOLUTION OF FIELD DEPLOYABLE fNIR SPECTROSCOPY FROM BENCH TO CLINICAL SETTINGS

In the late 1980s and early 1990s, Dr. Britton Chance and his colleagues, using picosecond-long laser pulses, spearheaded the development of time-resolved spectroscopy techniques in an effort to obtain quantitative information about the optical characteristics of the tissue. These efforts by Chance and colleagues expedited the translation of near-infrared spectroscopy (NIRS)-based techniques into a neuroimaging modality for various cognitive studies. Beginning in the early 2000s, Dr. Britton Chance guided and steered the collaboration with the Optical Brain Imaging team at Drexel University toward the development and application of a field deployable continuous wave functional near-infrared spectroscopy (fNIR) system as a means to monitor cognitive functions, particularly during attention and working memory tasks as well as for complex tasks such as war games and air traffic control scenarios performed by healthy volunteers under operational conditions. Further, these collaborative efforts led to various clinical applications, including traumatic brain injury, depth of anesthesia monitoring, pediatric pain assessment, and brain–computer interface in neurology. In this paper, we introduce how these collaborative studies have made fNIR an excellent candidate for specified clinical and research applications, including repeated cortical neuroimaging, bedside or home monitoring, the elicitation of a positive effect, and protocols requiring ecological validity. This paper represents a token of our gratitude to Dr. Britton Chance for his influence and leadership. Through this manuscript we show our appreciation by contributing to his commemoration and through our work we will strive to advance the field of optical brain imaging and promote his legacy.

Cognitive Workload and Learning Assessment During the Implementation of a Next-Generation Air Traffic Control Technology Using Functional Near-Infrared Spectroscopy

Neuroimaging technologies, such as functional near-infrared spectroscopy (fNIR), could provide performance metrics directly from brain-based measures to assess safety and performance of operators in high-risk fields. In this paper, we objectively and subjectively examine the cognitive workload of air traffic control specialists utilizing a next-generation conflict resolution advisory. Credible differences were observed between continuously increasing workload levels that were induced by increasing the number of aircraft under control. In higher aircraft counts, a possible saturation in brain activity was realized in the fNIR data. A learning effect was also analyzed across a three-day/nine-session training period. The difference between Day 1 and Day 2 was credible, while there was a noncredible difference between Day 2 and Day 3. The results presented in this paper indicate some advantages in objective measures of cognitive workload assessment with fNIR cortical imaging over the subjective workload assessment keypad.

An optical brain computer interface for environmental control

A brain computer interface (BCI) is a system that translates neurophysiological signals detected from the brain to supply input to a computer or to control a device. Volitional control of neural activity and its real-time detection through neuroimaging modalities are key constituents of BCI systems. The purpose of this study was to develop and test a new BCI design that utilizes intention-related cognitive activity within the dorsolateral prefrontal cortex using functional near infrared (fNIR) spectroscopy. fNIR is a noninvasive, safe, portable and affordable optical technique with which to monitor hemodynamic changes, in the brains cerebral cortex. Because of its portability and ease of use, fNIR is amenable to deployment in ecologically valid natural working environments. We integrated a control paradigm in a computerized 3D virtual environment to augment interactivity. Ten healthy participants volunteered for a two day study in which they navigated a virtual environment with keyboard inputs, but were required to use the fNIR-BCI for interaction with virtual objects. Results showed that participants consistently utilized the fNIR-BCI with an overall success rate of 84% and volitionally increased their cerebral oxygenation level to trigger actions within the virtual environment.

Measuring speaker–listener neural coupling with functional near infrared spectroscopy

The present study investigates brain-to-brain coupling, defined as inter-subject correlations in the hemodynamic response, during natural verbal communication. We used functional near-infrared spectroscopy (fNIRS) to record brain activity of 3 speakers telling stories and 15 listeners comprehending audio recordings of these stories. Listeners’ brain activity was significantly correlated with speakers’ with a delay. This between-brain correlation disappeared when verbal communication failed. We further compared the fNIRS and functional Magnetic Resonance Imaging (fMRI) recordings of listeners comprehending the same story and found a significant relationship between the fNIRS oxygenated-hemoglobin concentration changes and the fMRI BOLD in brain areas associated with speech comprehension. This correlation between fNIRS and fMRI was only present when data from the same story were compared between the two modalities and vanished when data from different stories were compared; this cross-modality consistency further highlights the reliability of the spatiotemporal brain activation pattern as a measure of story comprehension. Our findings suggest that fNIRS can be used for investigating brain-to-brain coupling during verbal communication in natural settings.