Silvio P. Eberhardt

OMAR a haptic display for speech perception by deaf and deaf-blind individuals

A system for haptic (i.e. kinesthetic and cutaneous) stimulation of the hand is described. While the immediate application involves display of speech information, a number of other man-machine interface applications may be feasible, including force-feedback devices for computer interaction and human pattern extraction from multiple datastreams. In an attempt to model more closely the information streams available via the Tadoma method, OMAR was developed to stimulate kinesthetic as well as tactile receptors, by moving and vibrating fingers in one or two dimensions using hard-disk head-positioning actuators. OMAR is being used in experiments involving basic haptic perception and supplementation of speechreading with haptic codings of speech correlates obtained via X-ray microbeam measurements.<<ETX>>

Single‐channel vibrotactile supplements to visual perception of intonation and stress

Two experiments were conducted to explore the effectiveness of a single vibrotactile stimulator to convey intonation (question versus statement) and contrastive stress (on one of the first three words of four 4- or 5-word sentences). In experiment I, artificially deafened normal-hearing subjects judged stress and intonation in counterbalanced visual-alone and visual-tactile conditions. Six voice fundamental frequency-to-tactile transformations were tested. Two sentence types were voiced throughout, and two contained unvoiced consonants. Benefits to speechreading were significant, but small. No differences among transformations were observed. In experiment II, only the tactile stimuli were presented. Significant differences emerged among the transformations, with larger differences for intonation than for stress judgments. Surprisingly, tactile-alone intonation identification was more accurate than visual-tactile for several transformations.

Competitive neural architecture for hardware solution to the assignment problem

Abstract A neural network architecture for competitive assignment is presented, with details of a very large scale integration (VLSI) design and characterization of critical circuits fabricated in complementary metal-oxide semiconductor (CMOS). The assignment problem requires that elements of two sets (e.g., resources and consumers) be associated with each other such as to minimize the total cost of the associations. Unlike previous neural implementations, association costs are applied locally to processing units (PUs, i.e., neurons), reducing connectivity to VLSI-compatible O(number of PUs). Also, each element in either set may be independently programmed to associate with one, several, or a range of elements of the other set. A novel method of “hysteretic annealing,” effected by gradually increasing positive feedback within each PU, was developed and compared in simulations to mean-field annealing implemented by increasing PU gain over time. The simulations (to size 64 × 64) consistently found optimal or near-optimal solutions, with settling times of about 150 microseconds, except for a few variable-gain annealing trials that exhibited oscillation.

Analog VLSI neural networks: implementation issues and examples in optimization and supervised learning

Time-critical neural network applications that require fully parallel hardware implementations for maximal throughput are considered. The rich array of technologies that are being pursued is surveyed, and the analog CMOS VLSI medium approach is focused on. This medium is messy in that limited dynamic range, offset voltages, and noise sources all reduce precision. The authors examine how neural networks can be directly implemented in analog VLSI, giving examples of approaches that have been pursued to date. Two important application areas are highlighted: optimization, because neural hardware may offer a speed advantage of orders of magnitude over other methods; and supervised learning, because of the widespread use and generality of gradient-descent learning algorithms as applied to feedforward networks. >

Analysis of arm trajectories of everyday tasks for the development of an upper-limb orthosis

Spatiotemporal arm and body movements of able-bodied subjects performing nine everyday tasks were recorded for the purpose of guiding the development of an upper-limb orthosis. To provide a user the opportunity to carry out these tasks with natural movements, the orthosis should allow replication of the measured trajectories. We outline the orthosis architecture, which supports the users upper arm and forearm, and analyze the movement data to obtain orthosis design specifications. Trajectories were obtained using six-degree-of-freedom magnetic position sensors affixed to the wrist, elbow, shoulder, trunk and head. Elbow trajectory data were decomposed into ranges along the principle Cartesian axes to provide a generally useful envelope measure. The smallest Cartesian parallel-piped that contained the elbow trajectories for most tasks was approximately 30 cm front/back, 15 cm side/side, and 17 cm up/down. A rough lower bound estimate obtained by asking subjects to repeat the tasks while minimizing elbow movement substantially reduced movement in the up/down and side/side dimensions. Elbow angles were generally in the range 50 degrees-150 degrees, and the angle of the forearm with respect to vertical was 10 degrees-110 degrees. Raw trajectory data may be downloaded from www://asel.udel.edu/robotics/orthosis/range.h tml.

Auditory Perceptual Learning for Speech Perception Can be Enhanced by Audiovisual Training

Speech perception under audiovisual (AV) conditions is well known to confer benefits to perception such as increased speed and accuracy. Here, we investigated how AV training might benefit or impede auditory perceptual learning of speech degraded by vocoding. In Experiments 1 and 3, participants learned paired associations between vocoded spoken nonsense words and nonsense pictures. In Experiment 1, paired-associates (PA) AV training of one group of participants was compared with audio-only (AO) training of another group. When tested under AO conditions, the AV-trained group was significantly more accurate than the AO-trained group. In addition, pre- and post-training AO forced-choice consonant identification with untrained nonsense words showed that AV-trained participants had learned significantly more than AO participants. The pattern of results pointed to their having learned at the level of the auditory phonetic features of the vocoded stimuli. Experiment 2, a no-training control with testing and re-testing on the AO consonant identification, showed that the controls were as accurate as the AO-trained participants in Experiment 1 but less accurate than the AV-trained participants. In Experiment 3, PA training alternated AV and AO conditions on a list-by-list basis within participants, and training was to criterion (92% correct). PA training with AO stimuli was reliably more effective than training with AV stimuli. We explain these discrepant results in terms of the so-called “reverse hierarchy theory” of perceptual learning and in terms of the diverse multisensory and unisensory processing resources available to speech perception. We propose that early AV speech integration can potentially impede auditory perceptual learning; but visual top-down access to relevant auditory features can promote auditory perceptual learning.

Speechreading sentences with single‐channel vibrotactile presentation of voice fundamental frequency

The main goal of this study was to investigate the efficacy of four vibrotactile speechreading supplements. Three supplements provided single-channel encodings of fundamental frequency (F0). Two encodings involved scaling and shifting glottal pulses to pulse rate ranges suited to tactual sensing capabilities; the third transformed F0 to differential amplitude of two fixed-frequency sinewaves. The fourth supplement added to one of the F0 encodings a second vibrator indicating high-frequency speech energy. A second goal was to develop improved methods for experimental control. Therefore, a sentence corpus was recorded on videodisc using two talkers whose speech was captured by video, microphone, and electroglottograph. Other experimental control issues included use of visual-alone control subjects, a multiple-baseline, single-subject design replicated for each of 15 normal-hearing subjects, sentence and syllable pre- and post-tests balanced for difficulty, and a speechreading screening test for subject selection. Across 17 h of treatment and 5 h of visual-alone baseline testing, each subject performed open-set sentence identification. Covariance analyses showed that the single-channel supplements provided a small but significant benefit, whereas the two-channel supplement was not effective. All subjects improved in visual-alone speechreading and maintained individual differences across the experiment. Vibrotactile benefit did not depend on speechreading ability.

A computational approach to analyzing sentential speech perception: phoneme-to-phoneme stimulus-response alignment.

A solution to the following problem is presented: Obtain a principled approach to studying error patterns in sentence-length responses obtained from subjects who were instructed to simply report what a talker had said. The solution is a sequence comparator that performs phoneme-to-phoneme alignment on transcribed stimulus and response sentences. Data for developing and testing the sequence comparator were obtained from 139 normal-hearing subjects who lipread (speechread) 100 sentences and from 15 different subjects who identified nonsense syllables by lipreading. Development of the sequence comparator involved testing two different costs metrics (visemes versus Euclidean distances) and two related comparison algorithms. After alignments with face validity were achieved, a validation experiment was conducted for which measures from random versus true stimulus-response sentence pairs were compared. Measures of phonemes correct and substitution uncertainty were found to be sensitive to the nature of the sentence pairs. In particular, correct phoneme matches were extremely rare in random pairings in comparison with true pairs. Also, an information-theoretic measure of uncertainty for substitutions in true versus random pairings showed that uncertainty was always higher for random than for true pairs.

Classroom Evaluation of the Arlyn Arm Robotic Workstation

High school and junior high school students with neuromuscular weakness and other disorders of the arms evaluated a recently commercialized robotic workstation, the Arlyn Arm, to carry out art projects and science experiments. These tasks were designed for independent execution with the workstation using standard or custom-designed tools. Each task was divided into subtasks, and the execution time of each subtask was determined as a measure of efficiency. Special attention was given to the causes of required experimenter intervention. While subjects easily accomplished some subtasks, others required considerable intervention. Most of these interventions could be avoided by further customizing accessories. It is concluded that the Arlyn Arm workstation could be of considerable benefit in a classroom setting to persons with severe neuromuscular disorders.

Audiovisual spoken word training can promote or impede auditory-only perceptual learning: prelingually deafened adults with late-acquired cochlear implants versus normal hearing adults

Training with audiovisual (AV) speech has been shown to promote auditory perceptual learning of vocoded acoustic speech by adults with normal hearing. In Experiment 1, we investigated whether AV speech promotes auditory-only (AO) perceptual learning in prelingually deafened adults with late-acquired cochlear implants. Participants were assigned to learn associations between spoken disyllabic C(=consonant)V(=vowel)CVC non-sense words and non-sense pictures (fribbles), under AV and then AO (AV-AO; or counter-balanced AO then AV, AO-AV, during Periods 1 then 2) training conditions. After training on each list of paired-associates (PA), testing was carried out AO. Across all training, AO PA test scores improved (7.2 percentage points) as did identification of consonants in new untrained CVCVC stimuli (3.5 percentage points). However, there was evidence that AV training impeded immediate AO perceptual learning: During Period-1, training scores across AV and AO conditions were not different, but AO test scores were dramatically lower in the AV-trained participants. During Period-2 AO training, the AV-AO participants obtained significantly higher AO test scores, demonstrating their ability to learn the auditory speech. Across both orders of training, whenever training was AV, AO test scores were significantly lower than training scores. Experiment 2 repeated the procedures with vocoded speech and 43 normal-hearing adults. Following AV training, their AO test scores were as high as or higher than following AO training. Also, their CVCVC identification scores patterned differently than those of the cochlear implant users. In Experiment 1, initial consonants were most accurate, and in Experiment 2, medial consonants were most accurate. We suggest that our results are consistent with a multisensory reverse hierarchy theory, which predicts that, whenever possible, perceivers carry out perceptual tasks immediately based on the experience and biases they bring to the task. We point out that while AV training could be an impediment to immediate unisensory perceptual learning in cochlear implant patients, it was also associated with higher scores during training.