Frédéric Dehais

Using near infrared spectroscopy and heart rate variability to detect mental overload

Mental workload is a key factor influencing the occurrence of human error, especially during piloting and remotely operated vehicle (ROV) operations, where safety depends on the ability of pilots to act appropriately. In particular, excessively high or low mental workload can lead operators to neglect critical information. The objective of the present study is to investigate the potential of functional near infrared spectroscopy (fNIRS) - a non-invasive method of measuring prefrontal cortex activity - in combination with measurements of heart rate variability (HRV), to predict mental workload during a simulated piloting task, with particular regard to task engagement and disengagement. Twelve volunteers performed a computer-based piloting task in which they were asked to follow a dynamic target with their aircraft, a task designed to replicate key cognitive demands associated with real life ROV operating tasks. In order to cover a wide range of mental workload levels, task difficulty was manipulated in terms of processing load and difficulty of control - two critical sources of workload associated with piloting and remotely operating a vehicle. Results show that both fNIRS and HRV are sensitive to different levels of mental workload; notably, lower prefrontal activation as well as a lower LF/HF ratio at the highest level of difficulty, suggest that these measures are suitable for mental overload detection. Moreover, these latter measurements point toward the existence of a quadratic model of mental workload.

Failure to Detect Critical Auditory Alerts in the Cockpit Evidence for Inattentional Deafness

Objective: The aim of this study was to test whether inattentional deafness to critical alarms would be observed in a simulated cockpit. Background: The inability of pilots to detect unexpected changes in their auditory environment (e.g., alarms) is a major safety problem in aeronautics. In aviation, the lack of response to alarms is usually not attributed to attentional limitations, but rather to pilots choosing to ignore such warnings due to decision biases, hearing issues, or conscious risk taking. Method: Twenty-eight general aviation pilots performed two landings in a flight simulator. In one scenario an auditory alert was triggered alone, whereas in the other the auditory alert occurred while the pilots dealt with a critical windshear. Results: In the windshear scenario, 11 pilots (39.3%) did not report or react appropriately to the alarm whereas all the pilots perceived the auditory warning in the no-windshear scenario. Also, of those pilots who were first exposed to the no-windshear scenario and detected the alarm, only three suffered from inattentional deafness in the subsequent windshear scenario. Conclusion: These findings establish inattentional deafness as a cognitive phenomenon that is critical for air safety. Pre-exposure to a critical event triggering an auditory alarm can enhance alarm detection when a similar event is encountered subsequently. Application: Case-based learning is a solution to mitigate auditory alarm misperception.

Real-Time State Estimation in a Flight Simulator Using fNIRS

Working memory is a key executive function for flying an aircraft. This function is particularly critical when pilots have to recall series of air traffic control instructions. However, working memory limitations may jeopardize flight safety. Since the functional near-infrared spectroscopy (fNIRS) method seems promising for assessing working memory load, our objective is to implement an on-line fNIRS-based inference system that integrates two complementary estimators. The first estimator is a real-time state estimation MACD-based algorithm dedicated to identifying the pilot’s instantaneous mental state (not-on-task vs. on-task). It does not require a calibration process to perform its estimation. The second estimator is an on-line SVM-based classifier that is able to discriminate task difficulty (low working memory load vs. high working memory load). These two estimators were tested with 19 pilots who were placed in a realistic flight simulator and were asked to recall air traffic control instructions. We found that the estimated pilot’s mental state matched significantly better than chance with the pilot’s real state (62% global accuracy, 58% specificity, and 72% sensitivity). The second estimator, dedicated to assessing single trial working memory loads, led to 80% classification accuracy, 72% specificity, and 89% sensitivity. These two estimators establish reusable blocks for further fNIRS-based passive brain computer interface development.

Why a Comprehensive Understanding of Mental Workload through the Measurement of Neurovascular Coupling Is a Key Issue for Neuroergonomics

Raja Parasuraman, the father of Neuroergonomics (the crossroads of Ergonomics and Neuroscience, Figure 1) has opened the doors to new discoveries and techniques for advancing understanding of human behavior with the underlying brain mechanisms (Parasuraman, 1998). As of his death in 2015, a precise and objective definition of the concept of mental workload (MWL) had still not yet been formulated. In this opinion piece, we posit that MWL is associated through the measurement of neurovascular coupling (NVC); innovative neuroimaging methods is now capable of measuring such a phenomenon; all while highlighting Parasuramans many contributions to this field.

Mitigation of Conflicts with Automation: Use of Cognitive Countermeasures

Objective: The aim of this study was to empirically assess the efficacy of cognitive countermeasures based on the technique of information removal to enhance human operator attentional disengagement abilities when facing attentional tunneling. Background: Lessons learned from human factors studies suggest that conflict with automation leads to the degradation of operators’ performance by promoting excessive focusing on a single task to the detriment of the supervision of other critical parameters. Method: An experimental setup composed of a real unmanned ground vehicle and a ground station was developed to test the efficiency of the cognitive countermeasures. The scenario (with and without countermeasure) involved an authority conflict between the participants and the robot induced by a battery failure. The effects of the conflict and, in particular, the impact of cognitive countermeasures on the participants’ cognition and arousal were assessed through heart rate measurement and eye tracking techniques. Results: In the control group (i.e., no countermeasure), 8 out of 12 participants experienced attentional tunneling when facing the conflict, leading them to neglect the visual alarms displayed that would have helped them to understand the evolution of the tactical situation. Participants in the countermeasure group showed lower heart rates and enhanced attentional abilities, and 10 out of 11 participants made appropriate decisions. Conclusions: The use of cognitive countermeasures appeared to be an efficient means to mitigate excessive focus issues in the unmanned ground vehicle environment. Applications: The principle of cognitive countermeasures can be applied to a large domain of applications involving human operators interacting with critical systems.

Formal Detection of Attentional Tunneling in Human Operator–Automation Interactions

The allocation of visual attention is a key factor for the humans when operating complex systems under time pressure with multiple information sources. In some situations, attentional tunneling is likely to appear and leads to excessive focus and poor decision making. In this study, we propose a formal approach to detect the occurrence of such an attentional impairment that is based on machine learning techniques. An experiment was conducted to provoke attentional tunneling during which psycho-physiological and oculomotor data from 23 participants were collected. Data from 18 participants were used to train an adaptive neuro-fuzzy inference system (ANFIS). From a machine learning point of view, the classification performance of the trained ANFIS proved the validity of this approach. Furthermore, the resulting classification rules were consistent with the attentional tunneling literature. Finally, the classifier was robust to detect attentional tunneling when performing over test data from four participants.

Attentional costs and failures in air traffic control notifications

Large display screens are common in supervisory tasks, meaning that alerts are often perceived in peripheral vision. Five air traffic control notification designs were evaluated in their ability to capture attention during an ongoing supervisory task, as well as their impact on the primary task. A range of performance measures, eye-tracking and subjective reports showed that colour, even animated, was less effective than movement, and notifications sometimes went unnoticed. Designs that drew attention to the notified aircraft by a pulsating box, concentric circles or the opacity of the background resulted in faster perception and no missed notifications. However, the latter two designs were intrusive and impaired primary task performance, while the simpler animated box captured attention without an overhead cognitive cost. These results highlight the need for a holistic approach to evaluation, achieving a balance between the benefits for one aspect of performance against the potential costs for another. Practitioner summary: We performed a holistic examination of air traffic control notification designs regarding their ability to capture attention during an ongoing supervisory task. The combination of performance, eye-tracking and subjective measurements demonstrated that the best design achieved a balance between attentional power and the overhead cognitive cost to primary task performance.

Reward and Uncertainty Favor Risky Decision-Making in Pilots: Evidence from Cardiovascular and Oculometric Measurements

In this paper we examined plan continuation error (PCE), a well known error made by pilots consisting in continuing the flight plan despite adverse meteorological conditions. Our hypothesis is that a large range of strong negative emotional consequences, including those induced by economic pressure, are associated with the decision to revise the flight plan and favor PCE. We investigated the economic hypothesis with a simplified landing task (reproduction of a real aircraft instrument) in which uncertainty and reward were manipulated. Heart rate (HR), heart rate variability (HRV) and eye tracking measurements were performed to get objective clues both on the cognitive and emotional state of the volunteers. Results showed that volunteers made more risky decisions under the influence of the financial incentive, in particular when uncertainty was high. Psychophysiological examination showed that HR increased and total HRV decreased in response to the cognitive load generated by the task. In addition, HR also increased in response to the financially motivated condition. Eventually, fixation times increased when uncertainty was high, confirming the difficulty in obtaining/interpreting information from the instrument in this condition. These results support the assumption that risky-decision making observed in pilots can be, at least partially, explained by a shift from cold to hot (emotional) decision-making in response to economic constraints and uncertainty.

Frequency analysis of a task-evoked pupillary response: Luminance-independent measure of mental effort

Pupil diameter is a widely-studied cognitive load measure, which, despite its convenience for non-intrusive operator state monitoring in complex environments, is still not available for in situ measurements because of numerous methodological limitations. The most important of these limitations is the influence of pupillary light reflex. Hence, there is the need of providing a pupil-based cognitive load measure that is independent of light conditions. In this paper, we present a promising technique of pupillary signal analysis resulting in luminance-independent measure of mental effort that could be used in real-time without a priori on luminous conditions. Twenty-two participants performed a short-term memory task under different screen luminance conditions. Our results showed that the amplitude of pupillary dilation due to load on memory was luminance-dependent with higher amplitude corresponding to lower-luminance condition. Furthermore, our experimentation showed that load on memory and luminance factors express themselves differently according to frequency. Therefore, as our statistical analysis revealed, the ratio between low (0-1.6 Hz) and high frequency (1.6-4 Hz) bands (LF/HF ratio) of power spectral densities of pupillary signal is sensitive to the cognitive load but not to luminance. Our results are promising for the measurement of load on memory in ecological settings.

Characterization of mind wandering using fNIRS

Assessing whether someone is attending to a task has become important for educational and professional applications. Such attentional drifts are usually termed mind wandering (MW). The purpose of the current study is to test to what extent a recent neural imaging modality can be used to detect MW episodes. Functional near infrared spectroscopy is a non-invasive neuroimaging technique that has never been used so far to measure MW. We used the Sustained Attention to Response Task (SART) to assess when subjects attention leaves a primary task. Sixteen-channel fNIRS data were collected over frontal cortices. We observed significant activations over the medial prefrontal cortex (mPFC) during MW, a brain region associated with the default mode network (DMN). fNIRS data were used to classify MW data above chance level. In line with previous brain-imaging studies, our results confirm the ability of fNIRS to detect Default Network activations in the context of MW.