Oana Balan

Designing sensory-substitution devices: Principles, pitfalls and potential1

An exciting possibility for compensating for loss of sensory function is to augment deficient senses by conveying missing information through an intact sense. Here we present an overview of techniques that have been developed for sensory substitution (SS) for the blind, through both touch and audition, with special emphasis on the importance of training for the use of such devices, while highlighting potential pitfalls in their design. One example of a pitfall is how conveying extra information about the environment risks sensory overload. Related to this, the limits of attentional capacity make it important to focus on key information and avoid redundancies. Also, differences in processing characteristics and bandwidth between sensory systems severely constrain the information that can be conveyed. Furthermore, perception is a continuous process and does not involve a snapshot of the environment. Design of sensory substitution devices therefore requires assessment of the nature of spatiotemporal continuity for the different senses. Basic psychophysical and neuroscientific research into representations of the environment and the most effective ways of conveying information should lead to better design of sensory substitution systems. Sensory substitution devices should emphasize usability, and should not interfere with other inter- or intramodal perceptual function. Devices should be task-focused since in many cases it may be impractical to convey too many aspects of the environment. Evidence for multisensory integration in the representation of the environment suggests that researchers should not limit themselves to a single modality in their design. Finally, we recommend active training on devices, especially since it allows for externalization, where proximal sensory stimulation is attributed to a distinct exterior object.

The Sound of Vision Project: On the Feasibility of an Audio-Haptic Representation of the Environment, for the Visually Impaired

The Sound of Vision project involves developing a sensory substitution device that is aimed at creating and conveying a rich auditory representation of the surrounding environment to the visually impaired. However, the feasibility of such an approach is strongly constrained by neural flexibility, possibilities of sensory substitution and adaptation to changed sensory input. We review evidence for such flexibility from various perspectives. We discuss neuroplasticity of the adult brain with an emphasis on functional changes in the visually impaired compared to sighted people. We discuss effects of adaptation on brain activity, in particular short-term and long-term effects of repeated exposure to particular stimuli. We then discuss evidence for sensory substitution such as Sound of Vision involves, while finally discussing evidence for adaptation to changes in the auditory environment. We conclude that sensory substitution enterprises such as Sound of Vision are quite feasible in light of the available evidence, which is encouraging regarding such projects.

Navigational audio games: an effective approach toward improving spatial contextual learning for blind people

Abstract For people who suffer from a visual impairment, navigating in an unknown environment represents a considerable challenge. To help them develop spatial orientation and mobility skills, researchers have employed audio-based computer games as a practical, interactive and user-centered learning approach. The highly immersive and attractive nature of audio games enables blind people to create a spatial representation of their environment that can be assigned to aid performing real-life navigation tasks. This paper reviews the most notable navigational audio-based games in terms of their conceptual and technological approach, accessibility and user interaction efficiency.

Time-consistent segmentation of indoor depth video frames

We propose a new method for segmenting indoor scenes captured with RGBD or depth cameras with a simple and fast iterative CPU segmentation. The segmentation consists of three main steps. In the first stage, regions grow by adding new pixels based on depth information and normal vectors. Next, multiple regions are merged into bigger surfaces based on adjacency and region statistics. The last stage of the method ensures interframe consistency based on a cost function between regions from subsequent frames. The planar segmentation can be used both for identifying indoor objects, which in general have planar faces, as well as for estimating the camera motion and then reconstructing the environment.

Perceptual feedback training for improving spatial acuity and resolving front-back confusion errors in virtual auditory environments

This paper aims to test the hypothesis that audiovisual perceptual feedback based training can improve the sound localization accuracy and the navigational skills in a virtual auditory display where 3D sounds are created using non-individualized HRTFs. In our experiment, we tested the spatial auditory localization performance of nine sighted subjects prior and consequently a series of training procedures based on audiovisual feedback. The results concluded that this cross-modal facilitation enhances spatial hearing, reduces the incidence of localization and front-back confusion errors and helps the users to navigate more accurately in the virtual auditory environments by improving their ability to detect 3D sound sources synthesized with non-individualized HRTFs.

A Framework for Activity Recognition Through Deep Learning and Abnormality Detection in Daily Activities

Activity recognition plays a key role in providing activity assistance and care for users in intelligent homes. This paper presents a two layer of convolutional neural networks to perform human action recognition using images provided by multiple cameras. We consider one PTZ camera and multiple Kinects in order to offer continuity over the users movement. The drawbacks of using only one type of sensor is minimized. For example, field of view provided by Kinect sensor is not wide enough to cover the entire room. Also, the PTZ camera is not able to detect and track a person in case of different situations, such as the person is sitting or it is under the camera. Also the system will identify abnormalities that can appear in sequences of performed daily activities. The system is tested in Ambient Intelligence Laboratory (AmI-Lab) at the University Politehnica of Bucharest.

Near Infrared Imaging for Tissue Analysis

Near infrared light or NIR is not absorbed well by water or blood, which are the main components of biological tissue. For this reason, it can reach a significant depth of optical penetration, thus achieving increased illumination of internal structures. Even if there are some approaches based on near infrared light that allow the analysis of tissue near the skin, there is very little research regarding the 3D reconstruction and visualization of NIR illuminated areas. The current paper presents a review of existing work, analyses the drawbacks and advantages of different methods and proposes new approaches to improve this imaging modality.

Usability assessment of assistive technology for blind and visually impaired

In this paper we describe the usability assessment of a system designed to help blind and visually impaired people to navigate and perceive the environment. The proposed system is based on sensory substitution, remapping the vision stimuli into audio and haptic ones. The goal of this study is to aid the development of the sensory substitution device (SSD) by understanding how the different choices in encoding and rendering the environmental information affects the users perception and experience while using the system. The preliminary results are presented to show the usability and usefulness of the proposed system.