Justin R. Estepp

The effects of day-to-day variability of physiological data on operator functional state classification.

The application of pattern classification techniques to physiological data has undergone rapid expansion. Tasks as varied as the diagnosis of disease from magnetic resonance images, brain-computer interfaces for the disabled, and the decoding of brain functioning based on electrical activity have been accomplished quite successfully with pattern classification. These classifiers have been further applied in complex cognitive tasks to improve performance, in one example as an input to adaptive automation. In order to produce generalizable results and facilitate the development of practical systems, these techniques should be stable across repeated sessions. This paper describes the application of three popular pattern classification techniques to EEG data obtained from asymptotically trained subjects performing a complex multitask across five days in one month. All three classifiers performed well above chance levels. The performance of all three was significantly negatively impacted by classifying across days; however two modifications are presented that substantially reduce misclassifications. The results demonstrate that with proper methods, pattern classification is stable enough across days and weeks to be a valid, useful approach.

Lack of association between human plasma oxytocin and interpersonal trust in a prisoner's dilemma paradigm

Expanding interest in oxytocin, particularly the role of endogenous oxytocin in human social behavior, has created a pressing need for replication of results and verification of assay methods. In this study, we sought to replicate and extend previous results correlating plasma oxytocin with trust and trustworthy behavior. As a necessary first step, the two most commonly used commercial assays were compared in human plasma via the addition of a known quantity of exogenous oxytocin, with and without sample extraction. Plasma sample extraction was found to be critical in obtaining repeatable concentrations of oxytocin. In the subsequent trust experiment, twelve samples in duplicate, from each of 82 participants, were collected over approximately six hours during the performance of a Prisoner’s Dilemma task paradigm that stressed human interpersonal trust. We found no significant relationship between plasma oxytocin concentrations and trusting or trustworthy behavior. In light of these findings, previous published work that used oxytocin immunoassays without sample extraction should be reexamined and future research exploring links between endogenous human oxytocin and trust or social behavior should proceed with careful consideration of methods and appropriate biofluids for analysis.

An assessment of non-stationarity in physiological cognitive state assessment using artificial neural networks

With increased attention toward physiological cognitive state assessment as a component in the larger field of applied neuroscience, the need to develop methods for robust, stable assessment of cognitive state has been expressed as critical to designing effective augmented human-machine systems. The technique of cognitive state assessment, as well as its benefits, has been demonstrated by many research groups. In an effort to move closer toward a realized system, efforts must now be focused on critical issues that remain unsolved, namely instability of pattern classifiers over the course of hours and days. This work, as part of the Cognitive State Assessment Competition 2011, seeks to explore methods for ‘learning’ non-stationarity as a mitigation for more generalized patterns that are stable over time courses that are not widely discussed in the literature.

The Impact of Video Compression on Remote Cardiac Pulse Measurement Using Imaging Photoplethysmography

Remote physiological measurement has great potential in healthcare and affective computing applications. Imaging photoplethysmography (iPPG) leverages digital cameras to recover the blood volume pulse from the human body. While the impact of video parameters such as resolution and frame rate on iPPG accuracy have been studied, there has not been a systematic analysis of video compression algorithms. We compared a set of commonly used video compression algorithms (x264 and x265) and varied the Constant Rate Factor (CRF) to measure pulse rate recovery for a range of bit rates (file sizes) and video qualities. We found that compression, even at a low CRF, degrades the blood volume pulse (BVP) signal-tonoise ratio considerably. However, the bit rate of a video can be substantially decreased (by a factor of over 1000) without destroying the BVP signal entirely. We found an approximately linear relationship between bit rate and BVP signal-to-noise ratio up to a CRF of 36. A faster decrease in SNR was observed for videos of the task involving larger head motions and the x265 algorithm appeared to work more effectively in these cases.

A practical method for quickly determining electrode positions in high-density EEG studies.

This report describes a simple and practical method for determining electrode positions in high-density EEG studies. This method reduces the number of electrodes for which accurate three-dimensional location must be measured, thus minimizing experimental set-up time and the possibility of digitization error. For each electrode cap, a reference data set is first established by placing the cap on a reference head and digitizing the 3-D position of each channel. A set of control channels are pre-selected that should be adequately distributed over the cap. A simple choice could be the standard 19 channels of the International 10-20 system or their closest substitutes. In a real experiment, only the 3-D positions of these control channels need to be measured and the position of each of the remaining channels are calculated from the position data of the same channels in the reference data set using a local transformation determined by the nearest three or four pairs of control channels. Six BioSemi ActiveTwo caps of different size and channel numbers were used to evaluate the method. Results show that the mean prediction error is about 2mm and is comparable with the residual uncertainty in direct position measurement using a Polhemus digitizer.

Physiological cognitive state assessment: Applications for designing effective human-machine systems

Significant growth in the field of neuroscience has occurred over the last decade such that new application areas for basic research techniques are opening up to practitioners in many other areas. Of particular interest to many is the principle of neuroergonomics, by which the traditional work in neuroscience and its related topics can be applied to non-traditional areas such as human-machine system design. While work in neuroergonomics certainly predates the use of the term in the literature (previously identified by others as applied neuroscience, operational neuroscience, etc.), there is great promise in the larger framework that is represented by the general context of the terminology. Here, we focus on the very specific concept that principles in brain-computer interfaces, neural prosthetics and the larger realm of machine learning using physiological inputs can be applied directly to the design and implementation of augmented human-machine systems. Indeed, work in this area has been ongoing for more than 25 years with very little cross-talk and collaboration between clinical and applied researchers. We propose that, given increased interest in augmented human-machine systems based on cognitive state, further progress will require research in the same vein as that being done in the aforementioned communities, and that all researchers with a vested interest in physiologically-based machine learning techniques can benefit from increased collaboration. We thereby seek to describe the current state of cognitive state assessment in human-machine systems, the problems and challenges faced, and the tightly-coupled relationship with other research areas. This supports the larger work of the Cognitive State Assessment 2011 Competition by setting the stage for the purpose of the session by showing the need to increase research in the machine learning techniques used by practitioners of augmented human-machine system design.

Fusing Partial Camera Signals for Noncontact Pulse Rate Variability Measurement

Remote camera-based measurement of physiology has great potential for healthcare and affective computing. Recent advances in computer vision and signal processing have enabled photoplethysmography (PPG) measurement using commercially available cameras. However, there remain challenges in recovering accurate noncontact PPG measurements in the presence of rigid head motion. When a subject is moving, their face may be turned away from one camera, be obscured by an object, or move out of the frame resulting in missing observations. As the calculation of pulse rate variability (PRV) requires analysis over a time window of several minutes, the effect of missing observations on such features is deleterious. We present an approach for fusing partial color-channel signals from an array of cameras that enable physiology measurements to be made from moving subjects, even if they leave the frame of one or more cameras, which would not otherwise be possible with only a single camera. We systematically test our method on subjects ( N=25) using a set of six, 5-min tasks (each repeated twice) involving different levels of head motion. This results in validation across 25 h of measurement. We evaluate pulse rate and PRV parameter estimation including statistical, geometric, and frequency-based measures. The median absolute error in pulse rate measurements was 0.57 beats-per-minute (BPM). In all but two tasks with the greatest motion, the median error was within 0.4 BPM of that from a contact PPG device. PRV estimates were significantly improved using our proposed approach compared to an alternative not designed to handle missing values and multiple camera signals; the error was reduced by over 50%. Without our proposed method, errors in pulse rate would be very high, and estimation of PRV parameters would not be feasible due to significant data loss.

Cross-Participant EEG-Based Assessment of Cognitive Workload Using Multi-Path Convolutional Recurrent Neural Networks

Applying deep learning methods to electroencephalograph (EEG) data for cognitive state assessment has yielded improvements over previous modeling methods. However, research focused on cross-participant cognitive workload modeling using these techniques is underrepresented. We study the problem of cross-participant state estimation in a non-stimulus-locked task environment, where a trained model is used to make workload estimates on a new participant who is not represented in the training set. Using experimental data from the Multi-Attribute Task Battery (MATB) environment, a variety of deep neural network models are evaluated in the trade-space of computational efficiency, model accuracy, variance and temporal specificity yielding three important contributions: (1) The performance of ensembles of individually-trained models is statistically indistinguishable from group-trained methods at most sequence lengths. These ensembles can be trained for a fraction of the computational cost compared to group-trained methods and enable simpler model updates. (2) While increasing temporal sequence length improves mean accuracy, it is not sufficient to overcome distributional dissimilarities between individuals’ EEG data, as it results in statistically significant increases in cross-participant variance. (3) Compared to all other networks evaluated, a novel convolutional-recurrent model using multi-path subnetworks and bi-directional, residual recurrent layers resulted in statistically significant increases in predictive accuracy and decreases in cross-participant variance.

Learning From the Slips of Others: Neural Correlates of Trust in Automated Agents

With the rise of increasingly complex artificial intelligence (AI), there is a need to design new methods to monitor AI in a transparent, human-aware manner. Decades of research have demonstrated that people, who are not aware of the exact performance levels of automated algorithms, often experience a mismatch in expectations. Consequently, they will often provide either too little or too much trust in an algorithm. Detecting such a mismatch in expectations, or trust calibration, remains a fundamental challenge in research investigating the use of automation. Due to the context-dependent nature of trust, universal measures of trust have not been established. Trust is a difficult construct to investigate because even the act of reflecting on how much a person trusts a certain agent can change the perception of that agent. We hypothesized that electroencephalograms (EEGs) would be able to provide such a universal index of trust without the need of self-report. In this work, EEGs were recorded for 21 participants (mean age = 22.1; 13 females) while they observed a series of algorithms perform a modified version of a flanker task. Each algorithm’s degree of credibility and reliability were manipulated. We hypothesized that neural markers of action monitoring, such as the observational error-related negativity (oERN) and observational error positivity (oPe), are potential candidates for monitoring computer algorithm performance. Our findings demonstrate that (1) it is possible to reliably elicit both the oERN and oPe while participants monitored these computer algorithms, (2) the oPe, as opposed to the oERN, significantly distinguished between high and low reliability algorithms, and (3) the oPe significantly correlated with subjective measures of trust. This work provides the first evidence for the utility of neural correlates of error monitoring for examining trust in computer algorithms.

Day-to-day variability in hybrid, passive brain-computer interfaces: Comparing two studies assessing cognitive workload

As hybrid, passive brain-computer interface systems become more advanced, it is important to grow our understanding of how to produce generalizable pattern classifiers of physiological data. One of the most difficult problems in applying machine learning algorithms to these data types is nonstationarity, which can evolve over the course of hours and days, and is more susceptible to changes resulting from complex cognitive function in comparison to simple, stimulus-based processes. This nonstationarity, referenced as day-to-day variability, results in the inability of many learning algorithms to generalize to new data. In previous work, we have shown that increasing the number of unique testing sessions used to form a learning set can improve the accuracy of classifying mental workload in a binary state paradigm. While this result was very promising, we did not address whether the additional discriminability was the result of a larger learning set or the uniqueness contributed by the testing sessions being spread over multiple days. Further, the simulation task used in this prior analysis was low-fidelity with respect to the task it attempted to model; whether these methods extend to more realistic task simulation environments has not been comparatively investigated. In this work, we compare these previous results to a second study, with a similar multi-day paradigm, that required participants to perform a more realistic simulation task. Comparative analysis of these two studies reveals that the improved generalization of the multi-day learning set is attributable, in large part, to the uniqueness of the multi-day paradigm. Further, this multi-day effect was also observed in the higher fidelity simulation study. These results help to validate the use of the multi-day learning set approach for improving overall system classification accuracy. Future studies should consider the use of multi-day designs for improving generalizability over other interesting dimensions.