Daniel Pacheco

XIM-engine: a software framework to support the development of interactive applications that uses conscious and unconscious reactions in immersive mixed reality

The development of systems that allow multimodal interpretation of human-machine interaction is crucial to advance our understanding and validation of theoretical models of user behavior. In particular, a system capable of collecting, perceiving and interpreting unconscious behavior can provide rich contextual information for an interactive system. One possible application for such a system is in the exploration of complex data through immersion, where massive amounts of data are generated every day both by humans and computer processes that digitize information at different scales and resolutions thus exceeding our processing capacity. We need tools that accelerate our understanding and generation of hypotheses over the datasets, guide our searches and prevent data overload. We describe XIM-engine, a bio-inspired software framework designed to capture and analyze multi-modal human behavior in an immersive environment. The framework allows performing studies that can advance our understanding on the use of conscious and unconscious reactions in interactive systems.

Spatializing experience: a framework for the geolocalization, visualization and exploration of historical data using VR/AR technologies

In this study we present a novel ICT framework for the exploration and visualization of historical information using Augmented Reality (AR) and geolocalization. The framework facilitates the geolocalization of multimedia files, as well as their later retrieval and visualization through an AR paradigm in which a virtual reconstruction is matched to users positions and viewing angle. The main objective of the architecture is to enhance human-data interaction with cultural heritage content in outdoor settings and generate more engaging and profound learning experiences by exploiting information spatialization and sequencing strategies.

Fast mental states decoding in mixed reality

The combination of Brain-Computer Interface (BCI) technology, allowing online monitoring and decoding of brain activity, with virtual and mixed reality (MR) systems may help to shape and guide implicit and explicit learning using ecological scenarios. Real-time information of ongoing brain states acquired through BCI might be exploited for controlling data presentation in virtual environments. Brain states discrimination during mixed reality experience is thus critical for adapting specific data features to contingent brain activity. In this study we recorded electroencephalographic (EEG) data while participants experienced MR scenarios implemented through the eXperience Induction Machine (XIM). The XIM is a novel framework modeling the integration of a sensing system that evaluates and measures physiological and psychological states with a number of actuators and effectors that coherently reacts to the users actions. We then assessed continuous EEG-based discrimination of spatial navigation, reading and calculation performed in MR, using linear discriminant analysis (LDA) and support vector machine (SVM) classifiers. Dynamic single trial classification showed high accuracy of LDA and SVM classifiers in detecting multiple brain states as well as in differentiating between high and low mental workload, using a 5 s time-window shifting every 200 ms. Our results indicate overall better performance of LDA with respect to SVM and suggest applicability of our approach in a BCI-controlled MR scenario. Ultimately, successful prediction of brain states might be used to drive adaptation of data representation in order to boost information processing in MR.

A Temporal Estimate of Integrated Information for Intracranial Functional Connectivity

A major challenge in computational and systems neuroscience concerns the quantification of information processing at various scales of the brain’s anatomy. In particular, using human intracranial recordings, the question we ask in this paper is: How can we estimate the informational complexity of the brain given the complex temporal nature of its dynamics? To address this we work with a recent formulation of network integrated information that is based on the Kullback-Leibler divergence between the multivariate distribution on the set of network states versus the corresponding factorized distribution over its parts. In this work, we extend this formulation for temporal networks and then apply it to human brain data obtained from intracranial recordings in epilepsy patients. Our findings show that compared to random re-wirings of the data, functional connectivity networks, constructed from human brain data, score consistently higher in the above measure of integrated information. This work suggests that temporal integrated information may indeed be a good starting point as a future measure of cognitive complexity.

Long-term spatial clustering in free recall

ABSTRACT We explored the influence of space on the organisation of items in long-term memory. In two experiments, we asked our participants to explore a virtual environment and memorise discrete items presented at specific locations. Memory for those items was later on tested in immediate (T1) and 24 hours delayed (T2) free recall tests, in which subjects were asked to recall as many items as possible in any order. In experiment 2, we further examined the contribution of active and passive navigation in recollection dynamics. Results across experiments revealed a significant tendency for participants to consecutively recall items that were encountered in proximate locations during learning. Moreover, the degree of spatial organisation and the total number of items recalled were positively correlated in the immediate and the delayed tests. Results from experiment 2 indicated that the spatial clustering of items was independent of navigation types. Our results highlight the long-term stability of spatial clustering effects and their correlation with recall performance, complementing previous results collected in immediate or briefly delayed tests.

A Spatial-Context Effect in Recognition Memory

We designed a novel experiment to investigate the modulation of human recognition memory by environmental context. Human participants were asked to navigate through a four-arm Virtual Reality (VR) maze in order to find and memorize discrete items presented at specific locations in the environment. They were later on tested on their ability to recognize items as previously presented or new. By manipulating the spatial position of half of the studied items during the testing phase of our experiment, we could assess differences in performance related to the congruency of environmental information at encoding and retrieval. Our results revealed that spatial context had a significant effect on the quality of memory. In particular, we found that recognition performance was significantly better in trials in which contextual information was congruent as opposed to those in which it was different. Our results are in line with previous studies that have reported spatial-context effects in recognition memory, further characterizing their magnitude under ecologically valid experimental conditions.