Oswald Barral

Predicting term-relevance from brain signals

Term-Relevance Prediction from Brain Signals (TRPB) is proposed to automatically detect relevance of text information directly from brain signals. An experiment with forty participants was conducted to record neural activity of participants while providing relevance judgments to text stimuli for a given topic. High-precision scientific equipment was used to quantify neural activity across 32 electroencephalography (EEG) channels. A classifier based on a multi-view EEG feature representation showed improvement up to 17% in relevance prediction based on brain signals alone. Relevance was also associated with brain activity with significant changes in certain brain areas. Consequently, TRPB is based on changes identified in specific brain areas and does not require user-specific training or calibration. Hence, relevance predictions can be conducted for unseen content and unseen participants. As an application of TRPB we demonstrate a high-precision variant of the classifier that constructs sets of relevant terms for a given unknown topic of interest. Our research shows that detecting relevance from brain signals is possible and allows the acquisition of relevance judgments without a need to observe any other user interaction. This suggests that TRPB could be used in combination or as an alternative for conventional implicit feedback signals, such as dwell time or click-through activity.

Covert Persuasive Technologies: Bringing Subliminal Cues to Human-Computer Interaction

The capability of machines to covertly persuade humans is both exciting and ethically concerning. In the present study we aim to bring subliminal masked stimulus paradigms to realistic environments, through Virtual Environments. The goal is to test if such paradigms are applicable to realistic setups while identifying the major challenges when doing so. We designed a study in which the user performed a realistic selection task in a virtual kitchen. For trials below one-second reaction time, we report significant effect of subliminal cues on the selection behavior. We conclude the study with a discussion of the challenges of bringing subliminal cueing paradigms to realistic HCI setups. Ethical concerns when designing covertly persuasive systems are discussed as well.

Natural brain-information interfaces: Recommending information by relevance inferred from human brain signals

Finding relevant information from large document collections such as the World Wide Web is a common task in our daily lives. Estimation of a user’s interest or search intention is necessary to recommend and retrieve relevant information from these collections. We introduce a brain-information interface used for recommending information by relevance inferred directly from brain signals. In experiments, participants were asked to read Wikipedia documents about a selection of topics while their EEG was recorded. Based on the prediction of word relevance, the individual’s search intent was modeled and successfully used for retrieving new relevant documents from the whole English Wikipedia corpus. The results show that the users’ interests toward digital content can be modeled from the brain signals evoked by reading. The introduced brain-relevance paradigm enables the recommendation of information without any explicit user interaction and may be applied across diverse information-intensive applications.

Applying Physiological Computing Methods to Study Psychological, Affective and Motivational Relevance

Relevance in information science has been studied for over forty years and robust frameworks have been derived. However, information retrieval systems are still using mainly objective, algorithmic measures of relevance. The aim of the present paper is to raise a discussion around the possibility that bring state-of-the-art physiological computing methods to model subjective components of relevance. We center the discussion on the relevance types known in the information science literature as psychological, affective and motivational relevance. The paper presents a definition of these concepts, as well as an overview of the recent advances in physiological computing methods developed in information science and information retrieval. We conclude with a discussion around the potential of physiological computing methods to model psychological, affective or motivational relevance.

Extracting relevance and affect information from physiological text annotation

We present physiological text annotation, which refers to the practice of associating physiological responses to text content in order to infer characteristics of the user information needs and affective responses. Text annotation is a laborious task, and implicit feedback has been studied as a way to collect annotations without requiring any explicit action from the user. Previous work has explored behavioral signals, such as clicks or dwell time to automatically infer annotations, and physiological signals have mostly been explored for image or video content. We report on two experiments in which physiological text annotation is studied first to (1) indicate perceived relevance and then to (2) indicate affective responses of the users. The first experiment tackles the user’s perception of relevance of an information item, which is fundamental towards revealing the user’s information needs. The second experiment is then aimed at revealing the user’s affective responses towards a -relevant- text document. Results show that physiological user signals are associated with relevance and affect. In particular, electrodermal activity was found to be different when users read relevant content than when they read irrelevant content and was found to be lower when reading texts with negative emotional content than when reading texts with neutral content. Together, the experiments show that physiological text annotation can provide valuable implicit inputs for personalized systems. We discuss how our findings help design personalized systems that can annotate digital content using human physiology without the need for any explicit user interaction.

How Semantic Processing of Words Evokes Changes in Pupil

This paper investigates the relationship between semantic processing of words and modifications in pupil size. Variations in pupil diameter reflect cognitive processing, as has been widely demonstrated in literature. We designed an experiment in which semantic association between words was manipulated in order to disclose potential differences in cognitive processing. Moreover, we measured the concurrent pupil diameter changes. Results showed faster pupil dilation in trials in which words were semantically associated. As changes in pupil diameter do not occur under voluntary control, they could reflect processing of preconscious information. We believe that a better symbiotic relationship between humans and machines is achievable once systems are able to make us aware of these “involuntary” changes.

Developing a Symbiotic System for Scientific Information Seeking: The Mindsee Project

This paper describes an approach for improving the current systems supporting the exploration and research of scientific literature, which generally adopt a query-based information-seeking paradigm. Our approach is to use a symbiotic system paradigm, exploiting central and peripheral physiological data along with eye-tracking data to adapt to users’ ongoing subjective relevance and satisfaction with search results. The system described, along with the interdisciplinary theoretical work underpinning it, could serve as a stepping stone for the development and diffusion of next-generation symbiotic systems, enabling a productive interdependence between humans and machines. After introducing the concept and evidence informing the development of symbiotic systems over a wide range of application domains, we describe the rationale of the MindSee project, emphasizing its BCI component and pinpointing the criteria around which users’ evaluations can gravitate. We conclude by summarizing the main contribution that MindSee is expected to make.

Integrating Neurophysiological Relevance Feedback in Intent Modeling for Information Retrieval

The use of implicit relevance feedback from neurophysiology could deliver effortless information retrieval. However, both computing neurophysiologic responses and retrieving documents are characterized by uncertainty because of noisy signals and incomplete or inconsistent representations of the data. We present the first‐of‐its‐kind, fully integrated information retrieval system that makes use of online implicit relevance feedback generated from brain activity as measured through electroencephalography (EEG), and eye movements. The findings of the evaluation experiment (N = 16) show that we are able to compute online neurophysiology‐based relevance feedback with performance significantly better than chance in complex data domains and realistic search tasks. We contribute by demonstrating how to integrate in interactive intent modeling this inherently noisy implicit relevance feedback combined with scarce explicit feedback. Although experimental measures of task performance did not allow us to demonstrate how the classification outcomes translated into search task performance, the experiment proved that our approach is able to generate relevance feedback from brain signals and eye movements in a realistic scenario, thus providing promising implications for future work in neuroadaptive information retrieval (IR).

BCI for Physiological Text Annotation

Automatic annotation of media content has become a critically important task for many digital services as the quantity of available online media content has grown exponentially. One approach is to annotate the content using the physiological responses of the media consumer. In the present paper, we reflect on three case studies that use brain signals for implicit text annotation to discuss the challenges faced when bringing passive brain-computer interfaces for physiological text annotation to the real world.

No Need to Laugh Out Loud: Predicting Humor Appraisal of Comic Strips Based on Physiological Signals in a Realistic Environment

We explore electroencephalography (EEG), electrodermal activity (EDA), and electrocardiography (ECG) as valid sources to infer humor appraisal in a realistic environment. We report on an experiment in which 25 participants browsed a popular user-generated humorous content website while their physiological responses were recorded. We build predictive models to infer the participants’ appraisal of the humorousness of the content and demonstrate that the fusion of several physiological signals can lead to classification performances up to 0.73 in terms of the area under the ROC curve (AUC). We identify that the most discriminative changes in physiological signals happen at the later stages of the information consumption process, reflected in changes on the upper EEG frequency bands, higher levels of EDA, and heart-rate acceleration. Additionally, we present a comprehensive analysis by benchmarking the predictive power of each of the physiological signals separately, and by comparing them to state-of-the-art facial recognition algorithms based on facial video recordings. The classification performance ranges from 0.88 (in terms of AUC) when combining physiological signals and video recordings, to 0.55 when using ECG signals alone.