Sébastien Scannella

Real-Time State Estimation in a Flight Simulator Using fNIRS

Working memory is a key executive function for flying an aircraft. This function is particularly critical when pilots have to recall series of air traffic control instructions. However, working memory limitations may jeopardize flight safety. Since the functional near-infrared spectroscopy (fNIRS) method seems promising for assessing working memory load, our objective is to implement an on-line fNIRS-based inference system that integrates two complementary estimators. The first estimator is a real-time state estimation MACD-based algorithm dedicated to identifying the pilot’s instantaneous mental state (not-on-task vs. on-task). It does not require a calibration process to perform its estimation. The second estimator is an on-line SVM-based classifier that is able to discriminate task difficulty (low working memory load vs. high working memory load). These two estimators were tested with 19 pilots who were placed in a realistic flight simulator and were asked to recall air traffic control instructions. We found that the estimated pilot’s mental state matched significantly better than chance with the pilot’s real state (62% global accuracy, 58% specificity, and 72% sensitivity). The second estimator, dedicated to assessing single trial working memory loads, led to 80% classification accuracy, 72% specificity, and 89% sensitivity. These two estimators establish reusable blocks for further fNIRS-based passive brain computer interface development.

P300 Event-Related Potential as an Indicator of Inattentional Deafness?

An analysis of airplane accidents reveals that pilots sometimes purely fail to react to critical auditory alerts. This inability of an auditory stimulus to reach consciousness has been coined under the term of inattentional deafness. Recent data from literature tends to show that tasks involving high cognitive load consume most of the attentional capacities, leaving little or none remaining for processing any unexpected information. In addition, there is a growing body of evidence for a shared attentional capacity between vision and hearing. In this context, the abundant information in modern cockpits is likely to produce inattentional deafness. We investigated this hypothesis by combining electroencephalographic (EEG) measurements with an ecological aviation task performed under contextual variation of the cognitive load (high or low), including an alarm detection task. Two different audio tones were played: standard tones and deviant tones. Participants were instructed to ignore standard tones and to report deviant tones using a response pad. More than 31% of the deviant tones were not detected in the high load condition. Analysis of the EEG measurements showed a drastic diminution of the auditory P300 amplitude concomitant with this behavioral effect, whereas the N100 component was not affected. We suggest that these behavioral and electrophysiological results provide new insights on explaining the trend of pilots’ failure to react to critical auditory information. Relevant applications concern prevention of alarms omission, mental workload measurements and enhanced warning designs.

Frequency analysis of a task-evoked pupillary response: Luminance-independent measure of mental effort

Pupil diameter is a widely-studied cognitive load measure, which, despite its convenience for non-intrusive operator state monitoring in complex environments, is still not available for in situ measurements because of numerous methodological limitations. The most important of these limitations is the influence of pupillary light reflex. Hence, there is the need of providing a pupil-based cognitive load measure that is independent of light conditions. In this paper, we present a promising technique of pupillary signal analysis resulting in luminance-independent measure of mental effort that could be used in real-time without a priori on luminous conditions. Twenty-two participants performed a short-term memory task under different screen luminance conditions. Our results showed that the amplitude of pupillary dilation due to load on memory was luminance-dependent with higher amplitude corresponding to lower-luminance condition. Furthermore, our experimentation showed that load on memory and luminance factors express themselves differently according to frequency. Therefore, as our statistical analysis revealed, the ratio between low (0-1.6 Hz) and high frequency (1.6-4 Hz) bands (LF/HF ratio) of power spectral densities of pupillary signal is sensitive to the cognitive load but not to luminance. Our results are promising for the measurement of load on memory in ecological settings.

Effects of the audiovisual conflict on auditory early processes.

Auditory alarm misperception is one of the critical events that lead aircraft pilots to an erroneous flying decision. The rarity of these alarms associated with their possible unreliability may play a role in this misperception. In order to investigate this hypothesis, we manipulated both audiovisual conflict and sound rarity in a simplified landing task. Behavioral data and event related potentials (ERPs) of thirteen healthy participants were analyzed. We found that the presentation of a rare auditory signal (i.e., an alarm), incongruent with visual information, led to a smaller amplitude of the auditory N100 (i.e., less negative) compared to the condition in which both signals were congruent. Moreover, the incongruity between the visual information and the rare sound did not significantly affect reaction times, suggesting that the rare sound was neglected. We propose that the lower N100 amplitude reflects an early visual-to-auditory gating that depends on the rarity of the sound. In complex aircraft environments, this early effect might be partly responsible for auditory alarm insensitivity. Our results provide a new basis for future aeronautic studies and the development of countermeasures.

Processing functional near infrared spectroscopy signal with a kalman filter to assess working memory during simulated flight

Working memory (WM) is a key executive function for operating aircraft, especially when pilots have to recall series of air traffic control instructions. There is a need to implement tools to monitor WM as its limitation may jeopardize flight safety. An innovative way to address this issue is to adopt a Neuroergonomics approach that merges knowledge and methods from Human Factors, System Engineering, and Neuroscience. A challenge of great importance for Neuroergonomics is to implement efficient brain imaging techniques to measure the brain at work and to design Brain Computer Interfaces (BCI). We used functional near infrared spectroscopy as it has been already successfully tested to measure WM capacity in complex environment with air traffic controllers (ATC), pilots, or unmanned vehicle operators. However, the extraction of relevant features from the raw signal in ecological environment is still a critical issue due to the complexity of implementing real-time signal processing techniques without a priori knowledge. We proposed to implement the Kalman filtering approach, a signal processing technique that is efficient when the dynamics of the signal can be modeled. We based our approach on the Boynton model of hemodynamic response. We conducted a first experiment with nine participants involving a basic WM task to estimate the noise covariances of the Kalman filter. We then conducted a more ecological experiment in our flight simulator with 18 pilots who interacted with ATC instructions (two levels of difficulty). The data was processed with the same Kalman filter settings implemented in the first experiment. This filter was benchmarked with a classical pass-band IIR filter and a Moving Average Convergence Divergence (MACD) filter. Statistical analysis revealed that the Kalman filter was the most efficient to separate the two levels of load, by increasing the observed effect size in prefrontal areas involved in WM. In addition, the use of a Kalman filter increased the performance of the classification of WM levels based on brain signal. The results suggest that Kalman filter is a suitable approach for real-time improvement of near infrared spectroscopy signal in ecological situations and the development of BCI.

N270 sensitivity to conflict strength and working memory: A combined ERP and sLORETA study

The event-related potential N270 component is known to be an electrophysiological marker of the supramodal conflict processing. However little is know about the factors that may modulate its amplitude. In particular, among all studies that have investigated the N270, little or no control of the conflict strength and of the load in working memory have been done leaving a lack in the understanding of this component. We designed a spatial audiovisual conflict task with simultaneous target and cross-modal distractor to evaluate the N270 sensitivity to the conflict strength (i.e., visual target with auditory distractor or auditory target with visual distractor) and the load in working memory (goal task maintenance with frequent change in the target modality). In a first session, participants had to focus on one modality for the target position to be considered (left-hand or right-hand) while the distractor could be at the same side (compatible) or at opposite side (incompatible). In a second session, we used the same set of stimuli as in the first session with an additional distinct auditory signal that clued the participants to frequently switch between the auditory and the visual targets. We found that (1) reaction times and N270 amplitudes for conflicting situations were larger within the auditory target condition compared to the visual one, (2) the increase in target maintenance effort led to equivalent increase of both reaction times and N270 amplitudes within all conditions and (3) the right dorsolateral prefrontal cortex current density was higher for both conflicting and active maintenance of the target situations. These results provide new evidence that the N270 component is an electrophysiological marker of the supramodal conflict processing that is sensitive to the conflict strength and that conflict processing and active maintenance of the task goal are two functions of a common executive attention system.

Auditory Alarm Misperception in the Cockpit: An EEG Study of Inattentional Deafness

Missing auditory alarms is a critical safety issue in many domains such as aviation. To investigate this phenomenon, we designed a scenario involving three flying scenarios corresponding to three different level of difficulty along with an oddball paradigm in a motion flight simulator. This preliminary study was conducted with one pilot equipped with a 32-channel EEG. The results shown that manipulating the three levels of task difficulty led respectively to rates of 0, 37, and

Moving Average Convergence Divergence filter preprocessing for real-time event-related peak activity onset detection: application to fNIRS signals

54\,\%