Paul Bach-y-Rita

Electrotactile and vibrotactile displays for sensory substitution systems

Sensory substitution systems provide their users with environmental information through a human sensory channel (eye, ear, or skin) different from that normally used or with the information processed in some useful way. The authors review the methods used to present visual, auditory, and modified tactile information to the skin and discuss present and potential future applications of sensory substitution, including tactile vision substitution (TVS), tactile auditory substitution, and remote tactile sensing or feedback (teletouch). The relevant sensory physiology of the skin, including the mechanisms of normal touch and the mechanisms and sensations associated with electrical stimulation of the skin using surface electrodes (electrotactile, or electrocutaneous, stimulation), is reviewed. The information-processing ability of the tactile sense and its relevance to sensory substitution is briefly summarized. The limitations of current tactile display technologies are discussed, and areas requiring further research for sensory substitution systems to become more practical are suggested.<<ETX>>

Sensory substitution and the human-machine interface.

Recent advances in the instrumentation technology of sensory substitution have presented new opportunities to develop systems for compensation of sensory loss. In sensory substitution (e.g. of sight or vestibular function), information from an artificial receptor is coupled to the brain via a human-machine interface. The brain is able to use this information in place of that usually transmitted from an intact sense organ. Both auditory and tactile systems show promise for practical sensory substitution interface sites. This research provides experimental tools for examining brain plasticity and has implications for perceptual and cognition studies more generally.

Cognition and Motor Processes

I Motor Control and Action Planning.- 1 Cognitivism and Future Theories of Action: Some Basic Issues.- 2 A Distributed Processing View of Human Motor Control.- 3 The Apraxias, Purposeful Motor Behavior, and Left-Hemisphere Function.- 4 A Motor-Program Editor.- 5 Eye Movement Control During Reading: The Effect of Word Units.- II Motor Contributions to Perception and Cognition.- 6 Motor Theories of Cognitive Structure: A Historical Review.- 7 Context Effects and Efferent Factors in Perception and Cognition.- 8 Saccadic Eye Movements and Visual Stability: Preliminary Considerations Towards a Cognitive Approach.- 9 Scanning and the Distribution of Attention: The Current Status of Herons Sensory-Motor Theory.- 10 The Relationship Between Motor Processes and Cognition in Tactile Vision Substitution.- III Mediating Structures and Operations Between Cognition and Action.- 11 Mechanisms of Voluntary Movement.- 12 Evaluation: The Missing Link Between Cognition and Action.- 13 Modes of Linkage Between Perception and Action.- 14 The Contribution of Vision-Based Imagery to the Acquisition and Operation of a Transcription Skill.- 15 Speech Production and Comprehension: One Lexicon or Two?.- IV Attention, Cognition, and Skilled Performance.- 16 S-Oh-R: Oh Stages! Oh Resources!.- 17 Automatic Processing: A Review of Recent Findings and a Plea for an Old Theory.- 18 Motor Learning as a Process of Structural Constriction and Displacement.- V Interactions Between Cognition and Action in Development.- 19 Cognition and Action in Development: A Tutorial Discussion.- 20 Biodynamic Structures, Cognitive Correlates of Motive Sets and the Development of Motives in Infants.- 21 Discontinuity in the Development of Motor Control in Children.- Author Index.

Seeing with the Brain

We see with the brain, not the eyes (Bach-y-Rita, 1972); images that pass through our pupils go no further than the retina. From there image information travels to the rest of the brain by means of coded pulse trains, and the brain, being highly plastic, can learn to interpret them in visual terms. Perceptual levels of the brain interpret the spatially encoded neural activity, modified and augmented by nonsynaptic and other brain plasticity mechanisms (Bach-y-Rita, 1972, 1995, 1999, in press). However, the cognitive value of that information is not merely a process of image analysis. Perception of the image relies on memory, learning, contextual interpretation (e.g., we perceive intent of the driver in the slight lateral movements of a car in front of us on the highway), cultural, and other social factors that are probably exclusively human characteristics that provide “qualia” (Bach-y-Rita, 1996b). This is the basis for our tactile vision substitution system (TVSS) studies that, starting in 1963, have demonstrated that visual information and the subjective qualities of seeing can be obtained tactually using sensory substitution systems.1 The description of studies with this system have been taken INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION, 15(2), 285–295 Copyright

Seeing with the skin

A system for converting an optical image into a tactile display has been evaluated to see what promise it has as a visual substitution system. After surprisingly little training, Ss are able to recognize common objects and to describe their arrangement in three-dimensional space. When given control of the sensing and imaging device, a television camera, Ss quickly achieve external subjective localization of the percepts. Limitations of the system thus far appear to be more a function of display resolution than limitations of the skin as a receptor surface. The acquisition of skill with the device has been remarkably similar for blind and sighted Ss.

Tactile sensory substitution studies.

Abstract: Forty years ago a project to explore late brain plasticity was initiated that was to lead into a broad area of sensory substitution studies. The questions at that time were: Can a person who has never seen learn to see as an adult? Is the brain sufficiently plastic to develop an entirely new sensory system? The short answer to both questions is yes, first clearly demonstrated in 1969 ( Bach‐y‐Rita et al., 1969 ). To reach that conclusion, it was first necessary to find a way to get visual information to the brain. That took many years and is still the most challenging aspect of the research and the development of practical sensory substitution and augmentation systems. The sensor array is not a problem: a TV camera for blind persons; an accelerometer for persons with vestibular loss; a microphone for deaf persons. These are common and fully developed devices. The problem is the brain‐machine interface (BMI). In this short report, only two substitution systems are discussed, vision and vestibular substitution.

CLOSING AN OPEN-LOOP CONTROL SYSTEM: VESTIBULAR SUBSTITUTION THROUGH THE TONGUE

The human postural coordination mechanism is an example of a complex closed-loop control system based on multisensory integration [9,10,13,14]. In models of this process, sensory data from vestibular, visual, tactile and proprioceptive systems are integrated as linearly additive inputs that drive multiple sensory-motor loops to provide effective coordination of body movement, posture and alignment [5-8, 10, 11]. In the absence of normal vestibular (such as from a toxic drug reaction) and other inputs, unstable posture occurs. This instability may be the result of noise in a functionally open-loop control system [9]. Nonetheless, after sensory loss the brain can utilize tactile information from a sensory substitution system for functional compensation [1-4, 12]. Here we have demonstrated that head-body postural coordination can be restored by means of vestibular substitution using a head-mounted accelerometer and a brain-machine interface that employs a unique pattern of electrotactile stimulation on the tongue. Moreover, postural stability persists for a period of time after removing the vestibular substitution, after which the open-loop instability reappears.

Computer-assisted motivating rehabilitation (CAMR) for institutional, home, and educational late stroke programs.

Abstract Based on our results during the last 25 years, we are developing late stroke computer-assisted motivating rehabilitation (CAMR) for the upper extremity. Evidence has been accumulating that functional gains are possible even many years after the damage. However, postacute rehabilitation must be motivating and related to real-life functional activities, or it may fail to enlist active participation. With CAMR programs, such as briefly reported here, instead of exercise, the patient is engaged in a game (e.g., ping-pong); instead of concentrating on the specific movements, he/she is concentrating on the game and the movements become subconscious. Patients, even those who initially consider that they cannot accomplish the task, show interest and improvement, and functional recovery appears to be extended beyond the specific movements that are being practiced. CAMR is also suitable for late functional reorganization programs in an educational model.

A 64-Solenoid, Four-Level Fingertip Search Display for the Blind

This paper describes the design and prototype model of a nonvibrating fingertip search display device. It was designed for use in experiments to determine the importance of fingertip exploration in tactile vision substitution. An 8 ×8 array of miniature dc solenoids mounted on 5 mm centers forms a raised two-dimensional display. With the aid of an IBM PC, it translates a visual image into a contour map of raised pins similar to transitory braille. The four possible pin heights of 0, 0.33, 0.67, and 1 mm represent discrete levels of image intensity. The user controls the location and resolution of the image sent from the IBM PC to the display by moving a mouse.

Electrotactile haptic display on the fingertips: preliminary results

Electrical stimulation of the sense of touch may be used to display pictorial information to blind computer users via a fingertip-scanned (haptic) touch tablet containing embedded electrodes. This might be particularly useful to users of systems with graphical user interfaces, or with drawing and layout software. Electrotactile (electrocutaneous) stimulation on the fingertip, however, differs substantially from that on other body locations. We discuss the implications of designing a practical graphics haptic display by addressing the high-resistance and variable nature of the fingertip electrode-skin interface.<<ETX>>