Thibault Gateau

Real-Time State Estimation in a Flight Simulator Using fNIRS

Working memory is a key executive function for flying an aircraft. This function is particularly critical when pilots have to recall series of air traffic control instructions. However, working memory limitations may jeopardize flight safety. Since the functional near-infrared spectroscopy (fNIRS) method seems promising for assessing working memory load, our objective is to implement an on-line fNIRS-based inference system that integrates two complementary estimators. The first estimator is a real-time state estimation MACD-based algorithm dedicated to identifying the pilot’s instantaneous mental state (not-on-task vs. on-task). It does not require a calibration process to perform its estimation. The second estimator is an on-line SVM-based classifier that is able to discriminate task difficulty (low working memory load vs. high working memory load). These two estimators were tested with 19 pilots who were placed in a realistic flight simulator and were asked to recall air traffic control instructions. We found that the estimated pilot’s mental state matched significantly better than chance with the pilot’s real state (62% global accuracy, 58% specificity, and 72% sensitivity). The second estimator, dedicated to assessing single trial working memory loads, led to 80% classification accuracy, 72% specificity, and 89% sensitivity. These two estimators establish reusable blocks for further fNIRS-based passive brain computer interface development.

Processing functional near infrared spectroscopy signal with a kalman filter to assess working memory during simulated flight

Working memory (WM) is a key executive function for operating aircraft, especially when pilots have to recall series of air traffic control instructions. There is a need to implement tools to monitor WM as its limitation may jeopardize flight safety. An innovative way to address this issue is to adopt a Neuroergonomics approach that merges knowledge and methods from Human Factors, System Engineering, and Neuroscience. A challenge of great importance for Neuroergonomics is to implement efficient brain imaging techniques to measure the brain at work and to design Brain Computer Interfaces (BCI). We used functional near infrared spectroscopy as it has been already successfully tested to measure WM capacity in complex environment with air traffic controllers (ATC), pilots, or unmanned vehicle operators. However, the extraction of relevant features from the raw signal in ecological environment is still a critical issue due to the complexity of implementing real-time signal processing techniques without a priori knowledge. We proposed to implement the Kalman filtering approach, a signal processing technique that is efficient when the dynamics of the signal can be modeled. We based our approach on the Boynton model of hemodynamic response. We conducted a first experiment with nine participants involving a basic WM task to estimate the noise covariances of the Kalman filter. We then conducted a more ecological experiment in our flight simulator with 18 pilots who interacted with ATC instructions (two levels of difficulty). The data was processed with the same Kalman filter settings implemented in the first experiment. This filter was benchmarked with a classical pass-band IIR filter and a Moving Average Convergence Divergence (MACD) filter. Statistical analysis revealed that the Kalman filter was the most efficient to separate the two levels of load, by increasing the observed effect size in prefrontal areas involved in WM. In addition, the use of a Kalman filter increased the performance of the classification of WM levels based on brain signal. The results suggest that Kalman filter is a suitable approach for real-time improvement of near infrared spectroscopy signal in ecological situations and the development of BCI.

Auditory Alarm Misperception in the Cockpit: An EEG Study of Inattentional Deafness

Missing auditory alarms is a critical safety issue in many domains such as aviation. To investigate this phenomenon, we designed a scenario involving three flying scenarios corresponding to three different level of difficulty along with an oddball paradigm in a motion flight simulator. This preliminary study was conducted with one pilot equipped with a 32-channel EEG. The results shown that manipulating the three levels of task difficulty led respectively to rates of 0, 37, and

Considering human's non-deterministic behavior and his availability state when designing a collaborative human-robots system

54\,\%

Disruption in neural phase synchrony is related to identification of inattentional deafness in real‐world setting

54% missed alarms. The EEG analyses revealed that this decrease in performance was associated with lower spectral power within the alpha band and reduced N100 component amplitude. This latter finding suggested the involvement of inattentional deafness mechanisms at an early stage of the auditory processing. Eventually, we implemented a processing chaini¾?to enhance the discriminability of ERPs for mental state monitoring purposes. The results indicated that this chain could be used in a quite ecological setting i.e. three-axis motion flight simulator as attested by the good results obtained for the oddball task, but also for more subtle mental states such as mental demand and stress level and the detection of target, that is to say the inattentional deafness phenomenon.

In silico vs. Over the Clouds: On-the-Fly Mental State Estimation of Aircraft Pilots, Using a Functional Near Infrared Spectroscopy Based Passive-BCI

Realizing long-term autonomous missions involving teams of heterogeneous robots is a challenge. It requires mechanisms to make robots react to disturbances or failures that will arise during the mission, while trying to successfully achieve the mission in cooperation. This paper presents HiDDeN, a distributed deliberative architecture that manages the execution of a hierarchical plan. This plan has initially been computed offline, ensuring some military operational constraints of the mission. Each robot’s supervisor then executes its own part of the plan, and reacts to failures using a hierarchical repair approach. This hierarchical repair has been designed with the sake of ensuring operational constraints, while reducing the need of communication between robots, as communication may be intermittent or even nonexistent when the robots operate in completely separate environments. HiDDeN’s robustness and scalability is evaluated with simulations. Experiments with an autonomous helicopter and an autonomous underwater vehicle have been realized and are presented as the defining point of our contribution.

EEG-Engagement Index and Auditory Alarm Misperception: An Inattentional Deafness Study in Actual Flight Condition

The objective of this study is to design a human-robots system that takes into account the non-deterministic nature of the human operators behavior. Such a system is implemented in a proof of concept scenario relying on a (MO)MDP decision framework that takes advantage of an eye-tracker device to estimate the cognitive availability of the human operator, and, some human operators inputs to deduce where he is focusing his attention. An experiment was conducted with ten participants interacting with a team of autonomous vehicles in a Search & Rescue scenario. Our results demonstrate the advantages of considering the cognitive availability of a human operator in such a complex context and also the interest of using such a decisional framework that can formally integrate the non-deterministic outcomes which model the human behavior.

Automation Surprise in Aviation: Real-Time Solutions

Real-time solutions for noise reduction and signal processing represent a central challenge for the development of Brain Computer Interfaces (BCI). In this paper, we introduce the Moving Average Convergence Divergence (MACD) filter, a tunable digital passband filter for online noise reduction and onset detection without preliminary learning phase, used in economic markets analysis. MACD performance was tested and benchmarked with other filters using data collected with functional Near Infrared Spectoscopy (fNIRS) during a digit sequence memorization task. This filter has a good performance on filtering and real-time peak activity onset detection, compared to other techniques. Therefore, MACD could be implemented for efficient BCI design using fNIRS.