Juan Diego Gomez

Toward local and global perception modules for vision substitution

Although retinal neural implants have considerably progressed they raise a number of questions concerning user acceptance, risk rejection, and cost. For the time being we support a low cost approach based on the transmission of limited vision information by means of the auditory channel. The See ColOr mobility aid for visually impaired individuals transforms a small portion of a coloured video image into sound sources represented by spatialised musical instruments. Basically, the conversion of colours into sounds is achieved by quantisation of the HSL colour system. Our purpose is to provide blind people with a capability of perception of the environment in real time. In this work the novelty is the simultaneous sonification of colour and depth, the last parameter being coded by sound rhythm. The main drawback of our approach is that the sonification of a limited portion of a captured image involves limited perception. As a consequence, we propose to extend the local perception module by introducing a new global perception module aiming at providing the user with a clear picture of the entire scene characteristics. Finally, we present several experiments to illustrate the limited perception module, such as: (1) detecting an open door in order to go out from the office; (2) walking in a hallway and looking for a blue cabinet; (3) walking in a hallway and looking for a red tee shirt; (4) avoiding two red obstacles; (5) moving outside and avoiding a parked car. Videos of experiments are available on http://www.youtube.com/guidobologna.

See ColOr: an extended sensory substitution device for the visually impaired

Purpose – The purpose of this paper is to overcome the limitations of sensory substitution methods (SSDs) to represent high-level or conceptual information involved in vision, which are mainly produced by the biological sensory mismatch between sight and substituting senses. Thus, provide the visually impaired with a more practical and functional SSD. Design/methodology/approach – Unlike any other approach, the SSD extends beyond a sensing prototype, by integrating computer vision methods to produce reliable knowledge about the physical world (at the lowest cost to the user). Importantly though, the authors do not abandon the typical encoding of low-level features into sound. The paper simply argues that any visual perception can be achieved through hearing needs to be reinforced or enhanced by techniques that lie beyond mere visual-to-audio mapping (e.g. computer vision, image processing). Findings – Experiments reported in this paper reveal that the See ColOr is learnable and functional, and provides ea...

Spatial awareness and intelligibility for the blind: audio-touch interfaces

Many people with visual disabilities mainly use audio feedback as a primary modality for interaction. Representing the visual environment with appropriate sounds contributes to make it intelligible to the blind. This audio-encoded environment still needs to be accessed in the same way as sighted people scan visual contents with their gaze. A finger-based scanning could be seen as a gaze-like strategy for those with visual impairments to be able of sensing an audio-represented context. We present in this work a computational interface that meets both, the visual-audio codification and the multi-touch interaction, so as to enlarge legibility of the environment for the blind and to facilitate navigating to desired locations, exploration, and serendipitous discovery. The core of this interface is the color and depth codification into musical instruments sounds, which effectively provides spatial awareness, audio revealing of boundaries and obstacles detection. The main contribution of our work is the assistance provided by this interface toward an active interaction of the user with his fingers that makes it possible to selectively explore, to discover points of interest, develop personalized strategies for navigating, and, in general, enjoy a greater sense of independence.

Toward 3D scene understanding via audio-description: Kinect-iPad fusion for the visually impaired

Microsofts Kinect 3-D motion sensor is a low cost 3D camera that provides color and depth information of indoor environments. In this demonstration, the functionality of this fun-only camera accompanied by an iPads tangible interface is targeted to the benefit of the visually impaired. A computer-vision-based framework for real time objects localization and for their audio description is introduced. Firstly, objects are extracted from the scene and recognized using feature descriptors and machine-learning. Secondly, the recognized objects are labeled by instruments sounds, whereas their position in 3D space is described by virtual space sources of sound. As a result, the scene can be heard and explored while finger-triggering the sounds within the iPad, on which a top-view of the objects is mapped. This enables blindfolded users to build a mental occupancy grid of the environment. The approach presented here brings the promise of efficient assistance and could be adapted as an electronic travel aid for the visually-impaired in the near future.

ViLimbs: Improving Phantom Limb Treatment Through Multisensory Feedback

Each year, anti-personnel mines contribute to a vast number of amputated victims due to the current conflict in Colombia. The recovery of such victims, includes psycho-motor therapies to reduce neuromuscular aftermaths, such as phantom limb pain (PLP). Therefore, improving rehabilitation strategies can potentially have a large positive impact in the recovery outcomes for long-term treatments. ViLimbs is a Virtual-Reality-based (VR) system aimed at enhancing classical mirror therapy for amputees using an immersive video-wall that renders the patient himself with a superimposed virtual limb. The patient-virtual-limb interaction is achieved by means of brain computer interfaces (BCI) and myoelectric signals read from the remaining part of the limb. Thus, voluntary movements get naturally transferred from patients to virtual limbs. Last but not least, patients sense of ownership over an alien virtual limb is enhanced through multisensory feedback (cardio-visual), which tends to lessen rehabilitation times.

Vision Substitution Experiments with See ColOr

See ColOr is a mobility aid for visually impaired people that uses the auditory channel to represent portions of captured images in real time. A distinctive feature of the See ColOr interface is the simultaneous coding of colour and depth. Four main modules were developed, in order to replicate a number of mechanisms present in the human visual systems. In this work, we first present the main experiments carried out in the first years of the project; among them : the avoidance of obstacles, the recognition and localization of objects, the detection of edges and the identification of coloured targets. Finally, we introduce new undergoing experiments in Colombia with blind persons, whose purpose is (1) to determine and to touch a target; (2) to navigate and to find a person; and (3) to find particular objects. Preliminary results illustrate encouraging results.

Efficient registering of color and range images

We present a simple yet highly efficient method to register range and color images. This method does not rely upon calibration parameters nor does it use visual features analysis. Our assumption is that if the transformation that registers the images is a mathematical function, we can approximate with little number of samples. To this end, thin-plate spline-based interpolations are used in this paper. Therefore, the registration of one point in our method takes only the solving of a nonlinear function. Drastically enhanced performances in the computational processing are attained under this condition. In fact, we show that ultimately our computational algorithm is independent of the complexity of the mathematical model underlying it. Finally, this paper reports on the results of experiments conducted with various range camera models that endorse the proposed method. Eventually, three key features can be derived from our method: practicality, accuracy, and wide applicability.

Non-visual-cueing-based sensing and understanding of nearby entities in aided navigation.

Exploring unfamiliar environments is a challenging task in which additionally, unsighted individuals frequently fail to gain perception of obstacles and make serendipitous discoveries. This is because the mental depiction of the context is drastically lessened due to the absence of visual information. It is still not clear in neuroscience, whether stimuli elicited by visual cueing can be replicated by other senses (cross-model transfer). In the practice, however, everyone recognizes a key, whether it is felt in a pocket or seen on a table. We present a context-aware aid system for the blind that merges three levels of assistance enhancing the intelligibility of the nearby entities: an exploration module to help gain awareness of the surrounding context, an alerting method for warning the user when a stumble is likely, and, finally, a recognition engine that retrieves natural targets previously learned. Practical experiences with our system show that in the absence of visual cueing, the audio and haptic trajectory playback coupled with computer-vision methods is a promising approach to depict dynamic information of the immediate environment.

Real-time image registration of RGB webcams and colorless 3d time-of-flight cameras

In line with the boom of 3D movies and cutting edge technologies, range cameras are increasingly common. Among others, time-of-flight (TOF) cameras give it the ability to capture three-dimensional images that reveal objects distances. A shortcoming of these sensors however, lies in that the majority does not provide color information (not even gray). Therefore they are useless in computer vision applications for which color is crucial. The PMD [vision] ® CamCube 3.0 is one example of an expensive colorless TOF camera. In this work, we attempt the addition of color to this camera by means of inexpensive resources. A regular webcam is stuck on top of the CamCube and its color images are registered into the TOF distance maps. To get this done, we developed an algorithm to enable real-time registration based solely on depth. Thus, this algorithm requires neither intrinsic parameters nor mono-calibration of none of the cameras. We finally show a tracking application in which a stumble is foretold if an object approaches following a threatening trajectory.