Doron Friedman

Self-paced (asynchronous) BCI control of a wheelchair in virtual environments: a case study with a tetraplegic

The aim of the present study was to demonstrate for the first time that brain waves can be used by a tetraplegic to control movements of his wheelchair in virtual reality (VR). In this case study, the spinal cord injured (SCI) subject was able to generate bursts of beta oscillations in the electroencephalogram (EEG) by imagination of movements of his paralyzed feet. These beta oscillations were used for a self-paced (asynchronous) brain-computer interface (BCI) control based on a single bipolar EEG recording. The subject was placed inside a virtual street populated with avatars. The task was to “go” from avatar to avatar towards the end of the street, but to stop at each avatar and talk to them. In average, the participant was able to successfully perform this asynchronous experiment with a performance of 90%, single runs up to 100%.

Portability by automatic translation: a large-scale case study

Abstract Constraint satisfaction problems are widely used in artificial intelligence. They involve finding values for problem variables subject to constraints that specify which combinations of values are consistent. Knowledge about properties of the constraints can permit inferences that reduce the cost of consistency checking. In particular, such inferences can be used to reduce the number of constraint checks required in establishing arc consistency, a fundamental constraint-based reasoning technique. A general AC-Inference algorithm schema is presented and various forms of inference discussed. A specific algorithm, AC-7, is presented, which takes advantage of a simple property common to all binary constraints to eliminate constraint checks that other arc consistency algorithms perform. The effectiveness of this approach is demonstrated analytically, and experimentally.Constraint satisfaction problems are widely used in artiicial intelligence. They involve nding values for problem variables subject to constraints that specify which combinations of values are consistent. Knowledge about properties of the constraints can permit inferences that reduce the cost of consistency checking. In particular, such inferences can be used to reduce the number of constraint checks required in establishing arc consistency, a fundamental constraint-based reasoning technique. A general AC-Inference algorithm schema is presented and various forms of inference discussed. A speciic algorithm, AC-7, is presented, which takes advantage of a simple property common to all binary constraints to eliminate constraint checks that other arc consistency algorithms perform. The eeectiveness of this approach is demonstrated analytically, and experimentally.

Spatial Social Behavior in Second Life

We have developed software bots that inhabit the popular online social environment SecondLife (SL). Our bots can wander around, collect data, engage in simple interactions, and carry out simple automated experiments. In this paper we use our bots to study spatial social behavior. We found an indication that SL users display distinct spatial behavior when interacting with other users. In addition, in an automated experiment carried out by our bot, we found that users, when their avatars were approached by our bot, tended to respond by moving their avatar, further indicating the significance of proxemics in SL.

Analysis of physiological responses to a social situation in an immersive virtual environment

An experiment was conducted in a Cave-like environment to explore the relationship between physiological responses and breaks in presence and utterances by virtual characters towards the participants. Twenty people explored a virtual environment (VE) that depicted a virtual bar scenario. The experiment was divided into a training and an experimental phase. During the experimental phase breaks in presence (BIPs) in the form of whiteouts of the VE scenario were induced for 2 s at four equally spaced times during the approximately 5 min in the bar scenario. Additionally, five virtual characters addressed remarks to the subjects. Physiological measures including electrocardiagram (ECG) and galvanic skin response (GSR) were recorded throughout the whole experiment. The heart rate, the heart rate variability, and the event-related heart rate changes were calculated from the acquired ECG data. The frequency response of the GSR signal was calculated with a wavelet analysis. The study shows that the heart rate and heart rate variability parameters vary significantly between the training and experimental phase. GSR parameters and event-related heart rate changes show the occurrence of breaks in presence. Event-related heart rate changes also signified the virtual character utterances. There were also differences in response between participants who report more or less socially anxious.

Immersive journalism: Immersive virtual reality for the first-person experience of news

This paper introduces the concept and discusses the implications of immersive journalism, which is the production of news in a form in which people can gain first-person experiences of the events or situation described in news stories. The fundamental idea of immersive journalism is to allow the participant, typically represented as a digital avatar, to actually enter a virtually recreated scenario representing the news story. The sense of presence obtained through an immersive system (whether a Cave or head-tracked head-mounted displays [HMD] and online virtual worlds, such as video games and online virtual worlds) affords the participant unprecedented access to the sights and sounds, and possibly feelings and emotions, that accompany the news. This paper surveys current approaches to immersive journalism and the theoretical background supporting claims regarding avatar experience in immersive systems. We also provide a specific demonstration: giving participants the experience of being in an interrogation room in an offshore prison. By both describing current approaches and demonstrating an immersive journalism experience, we open a new avenue for research into how presence can be utilized in the field of news and nonfiction.

Walking by thinking: the brainwaves are crucial, not the muscles!

Healthy participants are able to move forward within a virtual environment (VE) by the imagination of foot movement. This is achieved by using a brain-computer interface (BCI) that transforms thought-modulated electroencephalogram (EEG) recordings into a control signal. A BCI establishes a communication channel between the human brain and the computer. The basic principle of the Graz-BCI is the detection and classification of motor-imagery-related EEG patterns, whereby the dynamics of sensorimotor rhythms are analyzed. A BCI is a closed-loop system and information is visually fed back to the user about the success or failure of an intended movement imagination. Feedback can be realized in different ways, from a simple moving bar graph to navigation in VEs. The goals of this work are twofold: first, to show the influence of different feedback types on the same task, and second, to demonstrate that it is possible to move through a VE (e.g., a virtual street) without any muscular activity, using only the imagination of foot movement. In the presented work, data from BCI feedback displayed on a conventional monitor are compared with data from BCI feedback in VE experiments with a head-mounted display (HMD) and in a high immersive projection environment (Cave). Results of three participants are reported to demonstrate the proof-of-concept. The data indicate that the type of feedback has an influence on the task performance, but not on the BCI classification accuracy. The participants achieved their best performances viewing feedback in the Cave. Furthermore the VE feedback provided motivation for the subjects.

Navigating virtual reality by thought: what is it like?

We have set up a brain-computer interface (BCI) to be used as an input device to a highly immersive virtual reality CAVE-like system. We have carried out two navigation experiments: three subjects were required to rotate in a virtual bar room by imagining left or right hand movement, and to walk along a single axis in a virtual street by imagining foot or hand movement. In this paper we focus on the subjective experience of navigating virtual reality by thought, and on the interrelations between BCI and presence.

A method for generating an illusion of backwards time travel using immersive virtual reality—an exploratory study

We introduce a new method, based on immersive virtual reality (IVR), to give people the illusion of having traveled backwards through time to relive a sequence of events in which they can intervene and change history. The participant had played an important part in events with a tragic outcome—deaths of strangers—by having to choose between saving 5 people or 1. We consider whether the ability to go back through time, and intervene, to possibly avoid all deaths, has an impact on how the participant views such moral dilemmas, and also whether this experience leads to a re-evaluation of past unfortunate events in their own lives. We carried out an exploratory study where in the “Time Travel” condition 16 participants relived these events three times, seeing incarnations of their past selves carrying out the actions that they had previously carried out. In a “Repetition” condition another 16 participants replayed the same situation three times, without any notion of time travel. Our results suggest that those in the Time Travel condition did achieve an illusion of “time travel” provided that they also experienced an illusion of presence in the virtual environment, body ownership, and agency over the virtual body that substituted their own. Time travel produced an increase in guilt feelings about the events that had occurred, and an increase in support of utilitarian behavior as the solution to the moral dilemma. Time travel also produced an increase in implicit morality as judged by an implicit association test. The time travel illusion was associated with a reduction of regret associated with bad decisions in their own lives. The results show that when participants have a third action that they can take to solve the moral dilemma (that does not immediately involve choosing between the 1 and the 5) then they tend to take this option, even though it is useless in solving the dilemma, and actually results in the deaths of a greater number.

Centrally controlled heart rate changes during mental practice in immersive virtual environment: A case study with a tetraplegic

A tetraplegic patient was able to induce midcentral localized beta oscillations in the electroencephalogram (EEG) after extensive mental practice of foot motor imagery. This beta oscillation was used to simulate a wheel chair movement in a virtual environment (VE). The analysis of electrocardiogram (ECG) data revealed that the induced beta oscillations were accompanied by a characteristic heart rate (HR) change in form of a preparatory HR acceleration followed by a short-lasting deceleration in the order of 10-20 bpm (beats-per-minute). This provides evidence that mental practice of motor performance is accompanied not only by activation of cortical structures but also by central commands into the cardiovascular system with its nuclei in the brain stem.

Controlling an avatar by thought using real-time fMRI

OBJECTIVE We have developed a brain-computer interface (BCI) system based on real-time functional magnetic resonance imaging (fMRI) with virtual reality feedback. The advantage of fMRI is the relatively high spatial resolution and the coverage of the whole brain; thus we expect that it may be used to explore novel BCI strategies, based on new types of mental activities. However, fMRI suffers from a low temporal resolution and an inherent delay, since it is based on a hemodynamic response rather than electrical signals. Thus, our objective in this paper was to explore whether subjects could perform a BCI task in a virtual environment using our system, and how their performance was affected by the delay. APPROACH The subjects controlled an avatar by left-hand, right-hand and leg motion or imagery. The BCI classification is based on locating the regions of interest (ROIs) related with each of the motor classes, and selecting the ROI with maximum average values online. The subjects performed a cue-based task and a free-choice task, and the analysis includes evaluation of the performance as well as subjective reports. MAIN RESULTS Six subjects performed the task with high accuracy when allowed to move their fingers and toes, and three subjects achieved high accuracy using imagery alone. In the cue-based task the accuracy was highest 8-12 s after the trigger, whereas in the free-choice task the subjects performed best when the feedback was provided 6 s after the trigger. SIGNIFICANCE We show that subjects are able to perform a navigation task in a virtual environment using an fMRI-based BCI, despite the hemodynamic delay. The same approach can be extended to other mental tasks and other brain areas.