Geoffrey S. Meltzner

Measuring the neck frequency response function of laryngectomy patients: Implications for the design of electrolarynx devices

Measurements of the neck frequency response function (NFRF), defined as the ratio of the spectrum of the estimated volume velocity that excites the vocal tract to the spectrum of the acceleration delivered to the neck wall, were made at three different positions on the necks of nine laryngectomized subjects (five males and four females) and four normal laryngeal speakers (two males and two females). A minishaker driven by broadband noise provided excitation to the necks of subjects as they configured their vocal tracts to mimic the production of the vowels /a/, /ae/, and /I/. The sound pressure at the lips was measured with a microphone and an impedance head mounted on the shaker measured the acceleration. The neck wall passed low-frequency sound energy better than high-frequency sound energy, and thus the NFRF was accurately modeled as a low-pass filter. The NFRFs of the different subject groups (female laryngeal, male laryngeal speakers, laryngectomized males, and laryngectomized females) differed from each other in terms of corner frequency and gain, with both types of male subjects presenting NFRFs with larger overall gains. In addition, there was a notable amount of intersubject variability within groups. Because the NFRF is an estimate of how sound energy passes through the neck wall, these results should aid in the design of improved neck-type electrolarynx devices.

Sensor subset selection for surface electromyograpy based speech recognition

The authors previously reported speaker-dependent automatic speech recognition accuracy for isolated words using eleven surface-electromyographic (sEMG) sensors in fixed recording locations on the face and neck. The original array of sensors was chosen to ensure ample coverage of the muscle groups known to contribute to articulation during speech production. In this paper we systematically analyzed speech recognition performance from sensor subsets with the goal of reducing the number of sensors needed and finding the best combination of sensor locations to achieve word recognition rates comparable to the full set. We evaluated each of the different possible subsets by its mean word recognition rate across nine speakers using HMM modeling of MFCC and co-activation features derived from the subset of sensor signals. We show empirically that five sensors are sufficient to achieve a recognition rate to within a half a percentage point of that obtainable from the full set of sensors.

Pace independent mobile gait biometrics

Accelerometers embedded in mobile devices have shown great potential for non-obtrusive gait biometrics by directly capturing a users characteristic locomotion. Although gait analysis using these sensors has achieved highly accurate authentication and identification performance under controlled experimental settings, the robustness of such algorithms in the presence of assorted variations typical in real world scenarios remains a major challenge. In this paper, we propose a novel pace independent mobile gait biometrics algorithm that is insensitive to variability in walking speed. Our approach also exploits recent advances in invariant mobile gait representation to be independent of sensor rotation. Performance evaluations on a realistic mobile gait dataset containing 51 subjects confirm the merits of the proposed algorithm toward practical mobile gait authentication.

Signal processing advances for the MUTE sEMG-based silent speech recognition system

Military speech communication often needs to be conducted in very high noise environments. In addition, there are scenarios, such as special-ops missions, for which it is beneficial to have covert voice communications. To enable both capabilities, we have developed the MUTE (Mouthed-speech Understanding and Transcription Engine) system, which bypasses the limitations of traditional acoustic speech communication by measuring and interpreting muscle activity of the facial and neck musculature involved in silent speech production. This article details our recent progress on automatic surface electromyography (sEMG) speech activity detection, feature parameterization, multi-task sEMG corpus development, context dependent sub-word sEMG modeling, discriminative phoneme model training, and flexible vocabulary continuous sEMG silent speech recognition. Our current system achieved recognition accuracy at developable levels for a pre-defined special ops task. We further propose research directions in adaptive sEMG feature parameterization and data driven decision question generation for context-dependent sEMG phoneme modeling.

Integrating human and computer vision with EEG toward the control of a prosthetic arm

We are undertaking the development of a brain computer interface (BCI) [1] for control of an upper limb prosthetic. Our approach exploits electrical neural activity data for motor intent estimation, and eye gaze direction for target selection. These data streams are augmented by computer vision (CV) for 3D scene reconstruction, and are integrated with a hierarchical controller to achieve semi-autonomous control. User interfaces for the effective control of the many degrees of freedom (DOF) of advanced prosthetic arms are not yet available [2]. Ideally the combined arm and interface technology provides the user with reliable and dexterous capability for reaching, grasping and fine-scale manipulation. Technologies that improve arm embodiment i.e., the impression by the amputee subject that the arm is a natural part of their body-concept presents an important and difficult challenge to the human-robot interaction research community. Such embodiment is clearly predicated on cross-disciplinary advances, including accurate intent estimation and an and an algorithmic basis for natural arm control.

Silent Speech Recognition as an Alternative Communication Device for Persons With Laryngectomy

Each year thousands of individuals require surgical removal of the larynx (voice box) due to trauma or disease, and thereby require an alternative voice source or assistive device to verbally communicate. Although natural voice is lost after laryngectomy, most muscles controlling speech articulation remain intact. Surface electromyographic (sEMG) activity of speech musculature can be recorded from the neck and face, and used for automatic speech recognition to provide speech-to-text or synthesized speech as an alternative means of communication. This is true even when speech is mouthed or spoken in a silent (subvocal) manner, making it an appropriate communication platform after laryngectomy. In this study, eight individuals at least 6 months after total laryngectomy were recorded using eight sEMG sensors on their face (4) and neck (4) while reading phrases constructed from a 2500-word vocabulary. A unique set of phrases were used for training phoneme-based recognition models for each of the 39 commonly used phonemes in English, and the remaining phrases were used for testing word recognition of the models based on phoneme identification from running speech. Word error rates were on average 10.3% for the full eight-sensor set (averaging 9.5% for the top four participants), and 13.6% when reducing the sensor set to four locations per individual (n = 7). This study provides a compelling proof-of-concept for sEMG-based alaryngeal speech recognition, with the strong potential to further improve recognition performance.

Development of sEMG sensors and algorithms for silent speech recognition

OBJECTIVE Speech is among the most natural forms of human communication, thereby offering an attractive modality for human-machine interaction through automatic speech recognition (ASR). However, the limitations of ASR-including degradation in the presence of ambient noise, limited privacy and poor accessibility for those with significant speech disorders-have motivated the need for alternative non-acoustic modalities of subvocal or silent speech recognition (SSR). APPROACH We have developed a new system of face- and neck-worn sensors and signal processing algorithms that are capable of recognizing silently mouthed words and phrases entirely from the surface electromyographic (sEMG) signals recorded from muscles of the face and neck that are involved in the production of speech. The algorithms were strategically developed by evolving speech recognition models: first for recognizing isolated words by extracting speech-related features from sEMG signals, then for recognizing sequences of words from patterns of sEMG signals using grammar models, and finally for recognizing a vocabulary of previously untrained words using phoneme-based models. The final recognition algorithms were integrated with specially designed multi-point, miniaturized sensors that can be arranged in flexible geometries to record high-fidelity sEMG signal measurements from small articulator muscles of the face and neck. MAIN RESULTS We tested the system of sensors and algorithms during a series of subvocal speech experiments involving more than 1200 phrases generated from a 2200-word vocabulary and achieved an 8.9%-word error rate (91.1% recognition rate), far surpassing previous attempts in the field. SIGNIFICANCE These results demonstrate the viability of our system as an alternative modality of communication for a multitude of applications including: persons with speech impairments following a laryngectomy; military personnel requiring hands-free covert communication; or the consumer in need of privacy while speaking on a mobile phone in public.

Analysis of acoustic factors contributing to poor quality of electrolaryngeal speech

A large percentage of patients who have undergone laryngectomy to treat advanced laryngeal cancer rely on an electrolarynx (EL) to communicate orally. While serviceable, EL speech is plagued by shortcomings in both quality and intelligibility. As part of a larger ongoing effort to improve the oral communication of EL users, this study seeks to better quantify the sources and impacts of abnormal acoustic properties typically found in EL speech. Two EL characteristics, altered voicing source location and directly radiated self‐noise, were investigated by recording EL speech tokens in two recording conditions: (1) in an acoustic chamber (self‐noise present) and (2) through a specially constructed port into an acoustic chamber (self‐noise absent). An analysis of several vowels reveals that one or more of the first three formants are often masked by the direct noise and that their amplitudes are affected by EL location on the neck. Results are discussed relative to a comprehensive comparative analysis of pre‐l...